Flaky Tests, Test Maintenance and Testing in Production

Flakiness by the numbers

A flaky test passes and fails on the same code. The pipeline page introduces the idea; this page is about living with a large suite, where flakiness stops being an annoyance and becomes a property of the whole system. Google has published unusually concrete numbers on it:

• John Micco (2016): "about 1.5% of all test runs" gave a flaky result, "almost 16% of our tests have some level of flakiness", and "about 84% of the transitions we observe from pass to fail involve a flaky test". In other words, most of the time a test went from green to red, the code was not the reason.
• Jeff Listfield (2017): over a week, 0.5% of small tests, 1.6% of medium tests and 14% of large tests were flaky. Tests driving a browser or a phone emulator were flakier, but his conclusion was that "size is more predictive than tool": the more a test pulls in, the more can go wrong.

Note the different measures: 1.5% of runs, 16% of tests. Both matter, because a pipeline runs many tests at once. Kestrel's portal suite has about 1,200 tests. Suppose a dozen of them are flaky, each failing in 2% of runs for no reason:

def green_chance(flaky_tests, fail_rate, attempts=1):
    """Chance that a pipeline run is green when every flaky test fails
    independently with fail_rate per attempt, and gets `attempts` tries."""
    still_red = fail_rate ** attempts  # a test is red only if every try fails
    return (1 - still_red) ** flaky_tests


print("Flaky tests  red runs (no rerun)  red runs (1 rerun)")
for flaky in (1, 5, 12, 30):
    no_rerun = 1 - green_chance(flaky, 0.02)
    one_rerun = 1 - green_chance(flaky, 0.02, attempts=2)
    print(f"{flaky:>11}  {no_rerun:>19.1%}  {one_rerun:>18.2%}")

print()
# A real bug that breaks the test in half of all runs (a race condition)
for attempts in (1, 2, 3):
    print(f"Real 50% bug caught with {attempts} attempt(s): {0.5 ** attempts:.1%}")

Flaky tests  red runs (no rerun)  red runs (1 rerun)
          1                 2.0%               0.04%
          5                 9.6%               0.20%
         12                21.5%               0.48%
         30                45.5%               1.19%

Real 50% bug caught with 1 attempt(s): 50.0%
Real 50% bug caught with 2 attempt(s): 25.0%
Real 50% bug caught with 3 attempt(s): 12.5%

With twelve flaky tests, more than one pipeline run in five is red for nothing. Rerunning failed tests once makes the problem almost disappear, which is exactly why it is tempting. The last three lines show the price: a genuine intermittent bug, such as a race condition that breaks the code in half of all runs, is caught only a quarter of the time with one rerun. Reruns cannot tell a flaky test from a flaky product.

Causes and fixes

Flaky tests feel random, but their causes are a short, familiar list. Each has a standard fix:

Order-dependent tests

An order-dependent test passes or fails depending on which tests ran before it, because it relies on state that another test created or changed. It is easy to write by accident, and the full suite hides it: the test only fails when it runs alone, in another order, or in parallel on another worker. This module stores readings in a dictionary at module level:

cat > test_readings.py <<'EOF'
# A tiny in-memory reading store, shared by every test in the module
readings = {}


def add_reading(meter_id, kwh):
    readings.setdefault(meter_id, []).append(kwh)


def daily_total(meter_id):
    return round(sum(readings.get(meter_id, [])), 2)


def test_add_reading():
    add_reading("M-30001", 0.25)
    add_reading("M-30001", 0.40)
    assert len(readings["M-30001"]) == 2


def test_daily_total():
    # Quietly relies on the readings that test_add_reading left behind
    assert daily_total("M-30001") == 0.65
EOF

pytest -q                       # the whole file, in file order
pytest -q -k test_daily_total   # the same test, on its own

..                                                                       [100%]
2 passed in 0.16s
F                                                                        [100%]
=================================== FAILURES ===================================
_______________________________ test_daily_total _______________________________

    def test_daily_total():
        # Quietly relies on the readings that test_add_reading left behind
>       assert daily_total("M-30001") == 0.65
E       AssertionError: assert 0 == 0.65
E        +  where 0 = daily_total('M-30001')

test_readings.py:21: AssertionError
=========================== short test summary info ============================
FAILED test_readings.py::test_daily_total - AssertionError: assert 0 == 0.65
1 failed, 1 deselected in 0.16s

Nothing is random here, yet in a real pipeline it looks random: it breaks the day someone deletes or moves test_add_reading, runs one test on its own while debugging, or turns on parallel test runs that put the two tests on different workers. The fix is that every test gets a fresh store and creates its own data. A pytest fixture does that (see Unit Testing with pytest). Run on its own again, with pytest -q -k test_daily_total, the test now passes:

import pytest


class ReadingStore:
    def __init__(self):
        self.readings = {}

    def add(self, meter_id, kwh):
        self.readings.setdefault(meter_id, []).append(kwh)

    def daily_total(self, meter_id):
        return round(sum(self.readings.get(meter_id, [])), 2)


@pytest.fixture
def store():
    return ReadingStore()  # a new, empty store for every test


def test_add_reading(store):
    store.add("M-30001", 0.25)
    store.add("M-30001", 0.40)
    assert len(store.readings["M-30001"]) == 2


def test_daily_total(store):
    store.add("M-30001", 0.25)  # the test creates the data it needs
    store.add("M-30001", 0.40)
    assert store.daily_total("M-30001") == 0.65

.                                                                        [100%]
1 passed, 1 deselected in 0.17s

The same problem appears one level up with a shared test database: one test inserts customer C-10001, another counts customers and expects 3. Give each test its own transaction that is rolled back, or its own database or schema (see Integration and Contract Testing). Running the suite in random order now and then, with a plugin such as pytest-randomly (not installed here), makes these hidden dependencies surface early.

Fixed waits

The classic flaky end-to-end test waits a fixed time and hopes the page is ready. This one is from the Portal team's suite for the cost page:

import time

from playwright.sync_api import Page, expect


def test_night_cost_flaky(page: Page):
    page.goto("https://portal-test.kestrel.example.com/costs")
    page.get_by_role("button", name="This month").click()
    time.sleep(2)  # "the chart is always there after 2 seconds"
    assert page.get_by_test_id("night-cost").inner_text() == "€17.80"


def test_night_cost(page: Page):
    page.goto("https://portal-test.kestrel.example.com/costs")
    page.get_by_role("button", name="This month").click()
    # Retries until the text matches or the timeout (5 s by default) ends
    expect(page.get_by_test_id("night-cost")).to_have_text("€17.80")

Two seconds is usually enough on a laptop and sometimes not on a CI runner shared with other jobs, or on the day the test database is slower. Making it ten seconds makes the suite slow and only moves the problem. The second version waits for the condition itself and continues as soon as it is true. Playwright's assertions retry automatically; see End-to-End and UI Testing.

Time and dates

A test that calls date.today() or datetime.now() tests a different input every day. At Kestrel the obvious trap is daylight saving time: a test asserting that "today" has 96 quarter-hours passes all year and fails on 29 March and 25 October. Make time an input: pass the date or a clock object into the code, and write separate tests for the special days on purpose (see Test-Driven Development, where the 92- and 100-quarter-hour days drive the implementation). A pinned date also keeps the test meaningful in ten years.

Finding flaky tests

You cannot fix what you have not identified. Useful signals:

• Same commit, different result. A test that failed and then passed on the same commit is flaky by definition. CI systems and test reporting tools can record this per test.
• Repeat runs. Run a suspect test 50 or 100 times in a loop, on CI and not only locally. A failure rate of 1% needs many runs to show up.
• Run alone and shuffled. A test that only fails alone or in another order is order-dependent (above).
• Look at the failure, not just the color. A timeout in a wait, a "connection refused" or a missing element points to timing and environment, not to your change.

Automatic reruns

An automatic rerun repeats a failed test once or more within the same pipeline run and counts it as passed if a later attempt passes. Many CI systems and plugins offer it; for pytest it is the pytest-rerunfailures plugin (not installed on this site's test machine, so the output below is illustrative):

pytest -q --reruns 2 tests/e2e/

.....R.....                                                              [100%]
10 passed, 1 rerun in 41.37s

Reruns are a reasonable first-aid measure for end-to-end suites, on one condition: every rerun must be reported and counted. "1 rerun" in the summary is a flaky test announcing itself. A team that hides reruns behind a green check loses both the signal and, as the calculation above shows, most real intermittent bugs.

Quarantine

When a test turns out to be flaky and the cause is not fixed within the hour, take it out of the path that blocks merges, without losing it. A test quarantine moves known flaky tests into a separate, non-blocking part of the pipeline, where they still run and report, while each one is tracked as a bug with an owner and a deadline.

In pytest, a registered marker is the simplest mechanism: the blocking job deselects quarantined tests with -m "not quarantine", and a second job runs only them. The marker's reason carries the ticket:

cat > pyproject.toml <<'EOF'
[tool.pytest]
markers = [
    "quarantine: known flaky test with a ticket; does not block merges",
]
EOF

cat > test_costs.py <<'EOF'
import pytest


def night_rate(hour):
    return hour < 7 or hour >= 23


def test_23_is_night():
    assert night_rate(23)


def test_7_is_day():
    assert not night_rate(7)


def test_6_is_night():
    assert night_rate(6)


@pytest.mark.quarantine(reason="BUG-1046, owner Kwame Asante, fix by 2026-10-16")
def test_cost_page_shows_split():
    assert True  # stands in for the flaky end-to-end test
EOF

echo "--- blocking job"
pytest -q -m "not quarantine"
echo "--- quarantine job (allowed to fail)"
pytest -q -m quarantine

--- blocking job
...                                                                      [100%]
3 passed, 1 deselected in 0.41s
--- quarantine job (allowed to fail)
.                                                                        [100%]
1 passed, 3 deselected in 0.21s

In GitHub Actions the second job gets continue-on-error: true, so its failures show up in the run but do not fail the workflow or block a merge through a quality gate:

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: pip install -r requirements-dev.txt
      - run: pytest -m "not quarantine"

  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true # reported, never blocking
    steps:
      - uses: actions/checkout@v7
      - run: pip install -r requirements-dev.txt
      - run: pytest -m quarantine -rA

Kestrel's first quarantined test was the cost page test with the two-second sleep above. Kwame Asante replaced the sleep with a waiting assertion, ran it 100 times on CI without a failure, and moved it back. Because three more end-to-end tests had shown the same pattern, Amara Okafor, as problem manager, opened PRB-0720 ("Flaky end-to-end suite on the portal") to find the shared cause in the test environment, not just patch each test.

Test smells

A test smell is a sign in test code that something is wrong with the test's design, even if it currently passes. Like code smells, a smell is a reason to look, not proof of a problem. Gerard Meszaros catalogued many of them in xUnit Test Patterns (2007); his names are still common:

One smell not in every catalog deserves a mention: the test that cannot fail. An assertion on a value the test itself set, a mock that returns the expected answer, a try/except that swallows the failure. Mutation testing finds these systematically (see Code Coverage and Mutation Testing).

Maintaining a test suite

A test suite is code that the team has to read, change and run for years. Treat it that way:

• Review test code in pull requests as carefully as production code. A wrong expected value in a test is a bug that guards another bug.
• Refactor tests: shared setup into fixtures, test data into builders, repeated cases into parametrized tests.
• Delete tests that earn nothing: duplicates, tests for removed features, tests that check only that a mock was called. Fewer, clearer tests beat many vague ones.
• Update tests with behavior. When the variable tariff changed from €0.2500 to €0.2600 on 1 August 2026, the test that pinned the old price should fail, and the fix is to change the test together with the code, in the same pull request.
• Watch the suite's health: total duration, the slowest tests, the number of quarantined and skipped tests, and how often the main branch is red. Put these on the team's dashboard and discuss them in the Sprint Retrospective (see Quality in Agile Teams).

Testing in production

No test environment equals production. Real traffic has other volumes, other data, other browsers and other timing, and a few problems only appear there. Testing in production means deliberately checking a new version in the production environment, in a controlled way, in addition to (never instead of) testing before release. The techniques are covered elsewhere on this site:

For the release of the cost page (change CHG-0781), the Portal team plans: deploy with the feature flag off; switch it on for Kestrel staff for two days; then 5% of customers through the flag, watched like a canary: error rate and latency compared with the old page; then 25%, 50% and 100% if the numbers stay within the service level objective. The oracle in production is monitoring: without good metrics and logs, a canary is just a smaller outage.

• A kill switch. Every production test needs a way back that takes seconds: the flag off, traffic back to the old version.
• Limits agreed in advance. Which numbers stop the rollout, and who decides? The release plan's back-out criteria, or the error budget policy, is the place to write this down.
• No harm to customers or their data. Never test by changing real customers' contracts, readings or invoices. Use dedicated test accounts, and keep personal data out of test tooling.

Synthetic monitoring and real user monitoring

Synthetic monitoring runs scripted user journeys against the live system on a schedule, from outside, with test accounts: log in, open the cost page, check that the numbers are there. It is a form of black-box monitoring that goes beyond a single page load, and it is often the same end-to-end test, run against production every few minutes. Because it runs when no customer is active, it finds a broken journey at 03:00 instead of at the first customer's call at 08:00.

Kestrel runs a synthetic check of the cost page every five minutes as a test customer with meter M-TEST-0001. The check below shows the idea, against a small stand-in for the portal started inside the script. Note that it checks the content, not only the status code, and that it identifies itself with a header so the portal can keep synthetic traffic out of business numbers:

import json
import threading
import time
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

# A stand-in for the portal's cost API, with a switch to break it
state = {"broken": False}
requests_seen = {"customer": 0, "synthetic": 0}


class Portal(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass  # no request log lines in the output

    def do_GET(self):
        synthetic = self.headers.get("X-Synthetic-Check") is not None
        requests_seen["synthetic" if synthetic else "customer"] += 1
        parts = [] if state["broken"] else [
            {"part": "day", "eur": "41.12"},
            {"part": "night", "eur": "17.80"},
        ]
        body = json.dumps({"meter": "M-TEST-0001", "parts": parts}).encode()
        self.send_response(200)  # the bug still answers 200 OK
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


def check_cost_page(base_url):
    """One synthetic check: load the test customer's costs as a user would."""
    request = urllib.request.Request(
        base_url + "/api/costs/this-month",
        headers={"X-Synthetic-Check": "kestrel-synthetics"},
    )
    started = time.monotonic()
    with urllib.request.urlopen(request, timeout=5) as response:
        status = response.status
        data = json.load(response)
    seconds = time.monotonic() - started
    if status != 200:
        return f"FAIL: HTTP {status}"
    if sorted(part["part"] for part in data["parts"]) != ["day", "night"]:
        return "FAIL: HTTP 200, but the day/night split is missing"
    if seconds > 2.0:
        return "FAIL: slower than 2 seconds"
    return "OK"


server = ThreadingHTTPServer(("127.0.0.1", 0), Portal)
threading.Thread(target=server.serve_forever, daemon=True).start()
base = f"http://127.0.0.1:{server.server_port}"

print("09:00 check:", check_cost_page(base))
urllib.request.urlopen(base + "/api/costs/this-month").read()  # a real customer
state["broken"] = True  # a bad release goes out
print("09:05 check:", check_cost_page(base))
print("Requests seen:", requests_seen)
server.shutdown()

09:00 check: OK
09:05 check: FAIL: HTTP 200, but the day/night split is missing
Requests seen: {'customer': 1, 'synthetic': 2}

Real user monitoring (RUM) measures what actual users experience: a small script in the page reports load times, errors and the browser used, back to a collector. It covers every device and network that customers really have, which no synthetic check can, but it only sees journeys that users take, and only while they take them. The two complement each other: synthetic checks for "is it working right now?", RUM for "how is it for everybody?".