Code Coverage and Mutation Testing

Measuring how much testing you have: statement and branch coverage with coverage.py, why 100% coverage proves little, and mutation testing to check whether tests catch real changes.

What coverage measures

Code coverage is the share of the code that ran while the tests ran. The pipeline page builds a tiny line tracer by hand to show the idea; this page uses the real tool and goes one step further, to the question coverage cannot answer: would the tests notice if the code were wrong?

There are several ways to count "the code", and each is stricter than the one before:

  • Statement coverage (line coverage) counts the executable statements that ran at least once. It is the default in most tools and the number people usually mean when they say "we have 85% coverage".
  • Branch coverage counts the possible outcomes of every decision: an if has two (taken and not taken), a loop has two (run the body again or leave). 100% branch coverage means every decision went every way at least once.
  • Condition coverage and MC/DC look inside a decision, at the separate parts of a or b. They are covered further down; coverage.py does not measure them.

The ISTQB (International Software Testing Qualifications Board) Foundation syllabus treats statement testing and branch testing as the two white-box techniques every tester should know; the Test Design Techniques page puts them next to the black-box techniques.

Measuring coverage with coverage.py

coverage.py is the standard coverage tool for Python, maintained by Ned Batchelder (version 7.16.2 as of October 2026). It runs your program, usually the test runner, and records which lines and which jumps between lines executed. The pytest-cov plugin from the pipeline page is a thin wrapper around it. Using it directly takes two commands: coverage run collects the data into a file called .coverage, and coverage report prints the table.

The example writes a small tariff module from the Portal team's cost feature and two tests, then measures with branch coverage switched on: 

cat > tariff.py <<'EOF'
from datetime import time


def is_night(moment):
    return moment < time(7, 0) or moment >= time(23, 0)


def energy_cost(kwh, moment, tariff="day_night"):
    if kwh < 0:
        raise ValueError("kWh cannot be negative")
    if tariff == "fixed":
        price = 0.2700
    elif is_night(moment):
        price = 0.2240
    else:
        price = 0.3120
    return round(kwh * price, 2)
EOF
cat > test_tariff.py <<'EOF'
from datetime import time

from tariff import energy_cost


def test_day_rate():
    assert energy_cost(10, time(12, 0)) == 3.12


def test_night_rate():
    assert energy_cost(10, time(23, 30)) == 2.24
EOF
coverage run --branch -m pytest -q -p no:cacheprovider
coverage report -m
..                                                                       [100%]
2 passed in 0.05s
Name             Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------
tariff.py           12      2      6      2    78%   10, 12
test_tariff.py       6      0      0      0   100%
------------------------------------------------------------
TOTAL               18      2      6      2    83%

How to read the columns:

Column Meaning In this run
Stmts Executable statements in the file 12 in tariff.py
Miss Statements that never ran 2: line 10 (the raise) and line 12 (the fixed price)
Branch Possible branch outcomes 6: three if/elif tests, two ways each
BrPart Partial branches: decisions that went only one way 2: kwh < 0 was never true, tariff == "fixed" was never true
Cover Statements and branch outcomes combined 78%
Missing Line numbers that never ran (and, with partial branches whose target ran elsewhere, arcs such as 3->5) 10, 12

The report already says something useful: nobody tested a negative reading or the fixed tariff. It also shows a common annoyance: the test file itself is in the total, at 100%, which flatters the number. Tell coverage.py what your code is, in pyproject.toml, so the total means something: 

[tool.coverage.run]
branch = true
source = ["src"]

[tool.coverage.report]
show_missing = true
fail_under = 80

With fail_under set, coverage report ends with exit status 2 when the total is lower, which fails a pipeline step. The same settings work for pytest --cov.

Statement coverage versus branch coverage

The difference matters most for an if without an else. Take the first check of the reading validator (planned under change CHG-0761) and one test, for a negative reading: 

cat > validator.py <<'EOF'
def check_reading(kwh):
    problems = []
    if kwh < 0:
        problems.append("negative")
    return problems
EOF
cat > test_validator.py <<'EOF'
from validator import check_reading


def test_negative_reading_is_flagged():
    assert check_reading(-4.21) == ["negative"]
EOF
echo "--- statement coverage"
coverage run -m pytest -q -p no:cacheprovider
coverage report -m --include=validator.py
echo "--- branch coverage"
coverage run --branch -m pytest -q -p no:cacheprovider
coverage report -m --include=validator.py --fail-under=100
--- statement coverage
.                                                                        [100%]
1 passed in 0.03s
Name           Stmts   Miss  Cover   Missing
--------------------------------------------
validator.py       5      0   100%
--------------------------------------------
TOTAL              5      0   100%
--- branch coverage
.                                                                        [100%]
1 passed in 0.03s
Name           Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------
validator.py       5      0      2      1    86%   3->5
----------------------------------------------------------
TOTAL              5      0      2      1    86%
Coverage failure: total of 86 is less than fail-under=100

Every statement ran, so statement coverage is a perfect 100%. But the test never tried a reading that is not negative, so the jump from line 3 straight to line 5 (the condition is false, skip the append) never happened. Branch coverage reports that arc as 3->5. The percentage combines both counts: (5 statements run + 1 branch outcome taken) / (5 + 2) = 6/7 = 85.7%, shown as 86%. With --fail-under=100 the report fails, and the script ends with exit status 2.

The control flow of check_reading. Line 3 tests kwh less than zero. The true arc goes to line 4, append negative, and then to line 5, return. The false arc goes from line 3 straight to line 5. The test with -4.21 took the true arc and ran every line; the false arc from 3 to 5 was never taken. 3: if kwh < 0 4: append("negative") 5: return problems true false: 3->5 never taken by the test ran: lines 3, 4, 5 statements 5 of 5 branches 1 of 2
Statement coverage only sees the boxes; branch coverage also sees the arrows. The dashed arc is the untested case: a normal reading.

Conditions and MC/DC

coverage.py measures jumps between lines, not the parts of a condition. In is_night above, the whole rule moment < time(7, 0) or moment >= time(23, 0) is one line with no jump in it: one test at 23:30 makes the line "covered", even though the half that checks the morning was never decisive. Rewriting the rule over several lines with ifs changes the coverage number without changing the behavior, which says a lot about what the number means.

Stricter criteria exist for code where failure is not an option. Condition coverage asks that every single condition was both true and false. MC/DC (modified condition/decision coverage) also asks that each condition was shown to change the decision's outcome on its own, with the other conditions held fixed. For a or b that takes three tests: (false, false), (true, false) and (false, true). The aviation software standard DO-178C requires MC/DC for the most critical level of flight software (Level A), and other safety standards ask for it too; measuring it takes specialized tools. For a billing portal, branch coverage plus good boundary tests is the practical choice.

Why 100% coverage proves little

Coverage counts what ran, not what was checked. This test calls the validator with every kind of reading and asserts nothing; it can never fail, and the coverage report is perfect: 

cat > validator.py <<'EOF'
def check_reading(kwh):
    problems = []
    if kwh < 0:
        problems.append("negative")
    return problems
EOF
cat > test_validator.py <<'EOF'
from validator import check_reading


def test_all_kinds_of_readings():
    for kwh in (-4.21, 0.0, 0.4, 91.3):
        check_reading(kwh)  # no assert: this test checks nothing
EOF
coverage run --branch -m pytest -q -p no:cacheprovider
coverage report -m --include=validator.py
.                                                                        [100%]
1 passed in 0.03s
Name           Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------
validator.py       5      0      2      0   100%
----------------------------------------------------------
TOTAL              5      0      2      0   100%

Nobody writes a test like this on purpose. It happens when a coverage percentage becomes a target: "80% or the merge is blocked" gets met the cheapest way, by tests that call code without checking the result. That is Goodhart's law (when a measure becomes a target, it stops being a good measure) in a test suite. Coverage also cannot see:

  • Wrong expectations. A test that asserts the wrong price (say, the day rate at 23:00) covers the line and passes.
  • Missing code. If nobody wrote the check for a meter that went backwards, there is no line to be uncovered.
  • Boundaries. kwh < 0 and kwh <= 0 give exactly the same coverage with the tests above. Only a test with exactly 0 tells them apart.

Mutation testing: testing the tests

Mutation testing answers the question coverage cannot: if the code were slightly wrong, would a test fail? A tool makes one small change to the code, such as < into <=, + into -, 25 into 26 or True into False, and runs the tests. The changed version is a mutant. If at least one test fails, the mutant is killed: the tests noticed. If all tests still pass, the mutant survived: you have found a change in behavior that your suite does not care about. The tool repeats this for every change it can make, often hundreds per file.

The mutation testing loop. The original code is changed in one place to make a mutant, for example kwh less than 0 becomes kwh less than or equal to 0. The test suite runs against the mutant. If a test fails, the mutant is killed and the tests did their job. If all tests pass, the mutant survived, which points to a missing test. The tool then makes the next mutant. original code kwh < 0 mutant kwh <= 0 run tests a test fails killed: good all tests pass survived: a gap next change, next mutant
One mutant at a time: each change is small enough that a good test suite should notice it.

Why should such small changes tell you anything about real bugs? The idea goes back to DeMillo, Lipton and Sayward's 1978 paper "Hints on Test Data Selection", with two assumptions: programmers are mostly competent, so real bugs are small deviations from the right program (the competent programmer hypothesis), and tests that catch small mistakes tend to catch the bigger ones built from them too (the coupling effect). A > that should have been >=, the classic night-rate bug, is exactly such a small deviation.

Mutation testing with mutmut

For Python, the best-known tool is mutmut (3.8.0 as of October 2026). It expects the code in its own folder and the tests next to it, and reads its settings from pyproject.toml. Here is the full reading validator with three tests that look reasonable: a normal reading, a negative one where the meter also went backwards, and a 91.3 kWh spike like the one the analytics team found in September's daily totals (a decimal point in the wrong place). 

mkdir src tests
cat > pyproject.toml <<'EOF'
[tool.mutmut]
source_paths = ["src/"]

[tool.pytest]
pythonpath = ["src"]
EOF
cat > src/validator.py <<'EOF'
def check_reading(kwh, previous_total, new_total):
    """Return the problems found in one 15-minute reading."""
    problems = []
    if kwh < 0:
        problems.append("negative")
    if kwh > 25:
        problems.append("too high")
    if new_total < previous_total:
        problems.append("meter went backwards")
    return problems
EOF
cat > tests/test_validator.py <<'EOF'
from validator import check_reading


def test_normal_reading():
    assert check_reading(0.4, 1000.0, 1000.4) == []


def test_negative_reading():
    assert check_reading(-4.21, 1000.0, 995.79) == ["negative", "meter went backwards"]


def test_spike():
    assert check_reading(91.3, 1000.0, 1091.3) == ["too high"]
EOF
pytest -q -p no:cacheprovider
coverage run --branch -m pytest -q -p no:cacheprovider > /dev/null
coverage report --include="src/*"
mutmut run > /dev/null 2>&1
mutmut results
mutmut show validator.x_check_reading__mutmut_2
...                                                                      [100%]
3 passed in 0.04s
Name               Stmts   Miss Branch BrPart  Cover
----------------------------------------------------
src/validator.py       9      0      6      0   100%
----------------------------------------------------
TOTAL                  9      0      6      0   100%
    validator.x_check_reading__mutmut_2: survived
    validator.x_check_reading__mutmut_7: survived
    validator.x_check_reading__mutmut_8: survived
    validator.x_check_reading__mutmut_12: survived
# validator.x_check_reading__mutmut_2: survived
--- src/validator.py
+++ src/validator.py
@@ -1,7 +1,7 @@
 def check_reading(kwh, previous_total, new_total):
     """Return the problems found in one 15-minute reading."""
     problems = []
-    if kwh < 0:
+    if kwh <= 0:
         problems.append("negative")
     if kwh > 25:
         problems.append("too high")

Full branch coverage, every test green, and still four mutants survive. mutmut run made 15 mutants and the tests killed 11. (Its own progress output is a live counter with emoji, so the example hides it and asks for the results afterward.) mutmut results lists only the mutants that were not killed; mutmut show prints one as a diff against the original. Shown together, the four survivors are:

Mutant Change Missing test
mutmut_2 kwh < 0 → kwh <= 0 a reading of exactly 0 kWh (no consumption is valid)
mutmut_7 kwh > 25 → kwh >= 25 a reading of exactly 25 kWh (still allowed)
mutmut_8 kwh > 25 → kwh > 26 a reading just above 25, such as 25.5
mutmut_12 new_total < previous_total → <= a reading where the total stays the same

Every survivor is a missing boundary value. The tests picked comfortable values in the middle of each partition (0.4, −4.21, 91.3) and never touched the edges, which is exactly where off-by-one mistakes such as Priya's hour > 23 live. Coverage could not see this; mutation testing points at each gap by line and operator.

Killing the survivors

Two boundary tests, written straight from the table, kill three of the four: 

mkdir src tests
cat > pyproject.toml <<'EOF'
[tool.mutmut]
source_paths = ["src/"]

[tool.pytest]
pythonpath = ["src"]
EOF
cat > src/validator.py <<'EOF'
def check_reading(kwh, previous_total, new_total):
    """Return the problems found in one 15-minute reading."""
    problems = []
    if kwh < 0:
        problems.append("negative")
    if kwh > 25:
        problems.append("too high")
    if new_total < previous_total:
        problems.append("meter went backwards")
    return problems
EOF
cat > tests/test_validator.py <<'EOF'
from validator import check_reading


def test_normal_reading():
    assert check_reading(0.4, 1000.0, 1000.4) == []


def test_negative_reading():
    assert check_reading(-4.21, 1000.0, 995.79) == ["negative", "meter went backwards"]


def test_spike():
    assert check_reading(91.3, 1000.0, 1091.3) == ["too high"]


def test_zero_is_valid_and_total_may_stay_the_same():
    assert check_reading(0.0, 1000.0, 1000.0) == []


def test_exactly_25_kwh_is_allowed():
    assert check_reading(25.0, 1000.0, 1025.0) == []
EOF
mutmut run > /dev/null 2>&1
mutmut results
mutmut show validator.x_check_reading__mutmut_8
    validator.x_check_reading__mutmut_8: survived
# validator.x_check_reading__mutmut_8: survived
--- src/validator.py
+++ src/validator.py
@@ -3,7 +3,7 @@
     problems = []
     if kwh < 0:
         problems.append("negative")
-    if kwh > 25:
+    if kwh > 26:
         problems.append("too high")
     if new_total < previous_total:
         problems.append("meter went backwards")

The last survivor, kwh > 26, needs a value between 25 and 26 that must be flagged. Adding check_reading(25.5, 1000.0, 1025.5) == ["too high"] kills it too. Whether the tests should be that precise is a judgment call: if "more than 25 kWh in a quarter-hour" is a business rule that Joost Bakker's billing team relies on, yes.

The mutation score

The mutation score is the share of mutants the tests killed: here 11 of 15 (73%) before the new tests and 14 of 15 (93%) after. Tools differ in the details: some leave out mutants that timed out or that no test even reached, and some subtract the equivalent mutants described below. For pipelines, mutmut export-cicd-stats writes the counts to mutants/mutmut-cicd-stats.json.

Treat the score like coverage: a trend and a pointer, not a target. A survivor list you read is worth more than a percentage you report.

Equivalent mutants

Some mutants cannot be killed, because the change does not alter what the program does. Such an equivalent mutant is not a gap in the tests, and no test you write will ever fail on it. Two typical cases: 

# 1. The loop variable only ever grows by one, so < and != behave the same
i = 0
while i < len(readings):     # mutant: while i != len(readings):
    total += readings[i]
    i += 1

# 2. A value that is overwritten before anyone reads it
status = "new"               # mutant: status = "XXnewXX"
status = classify(reading)

Deciding that a mutant is equivalent takes a human: in general it is undecidable for a program to tell whether two programs always behave the same. Equivalent mutants are the main reason a score of 100% is often impossible and why survivors must be reviewed, not just counted. If one keeps coming back, mutmut lets you exclude the line with a # pragma: no mutate comment; write down why next to it.

Coverage and mutation testing in practice

Both tools are cheap to start and easy to misuse. What works for most teams:

  • Measure branch coverage in every pipeline run and show it in the pull request, ideally for the changed lines. A falling number on new code is a conversation; a fixed company-wide percentage is a game. The pipeline page shows a quality gate that blocks drops instead of demanding a level.
  • Run mutation testing where it pays. It runs the test suite once per mutant (mutmut uses one worker per processor core and only reruns the tests that cover the mutated function, but it is still slow). Run it nightly, or only on the files a pull request changed, and start with the code where a silent bug is expensive: pricing rules, validators, permission checks.
  • Read survivors at the three amigos table. A survivor such as kwh > 26 is often a question for the business ("is exactly 25 allowed?"), not only for the developer; see Quality in Agile Teams.
  • Delete the cache when tests change. mutmut keeps its results in mutants/; after editing tests, remove that folder (rm -rf mutants) before a new run, or you may see results of the old tests.

Mutation tools for other languages, as of October 2026:

Language Tool Version
Python mutmut; cosmic-ray 3.8.0; 8.7.0
Java PIT (Pitest) 1.30.0
JavaScript and TypeScript StrykerJS 10.0.0
C# and .NET Stryker.NET 5.0.0

For coverage in other languages, see the testing pages of Java, C#, JavaScript and PHP. Property-based tests, which invent their own boundary values, are another way to kill survivors: see Property-Based Testing.