End-to-End and UI Testing

What end-to-end tests are for

An end-to-end test (E2E test) checks a complete user journey through the running system: a browser opens the portal, logs in, clicks to the cost page and reads what the customer would read. On the way it passes through everything: the HTML (HyperText Markup Language) and JavaScript, the PHP portal, meter-api, the database and the configuration that connects them. That is its strength: a missing environment variable, a broken login redirect or a JavaScript error on one browser shows up here and nowhere else.

It is also its weakness. Because the test touches everything, it can fail for reasons that have nothing to do with the change under test: a slow test environment, another team's deployment, a cookie banner. Google measured this on its own tests: over a week, 0.5% of small tests showed flakiness, against 14% of large ones, and tests that drive a browser through WebDriver were among the flakiest (Jeff Listfield, Google Testing Blog, 2017). His conclusion: "Size is more predictive than tool."

So the Portal team keeps a short list. For the "your costs this month" feature (PBI-450) there is one end-to-end test: log in as a test customer with a day/night contract, open the cost page, and check that the 23:15 reading is billed at the night rate. The 06:59, 07:00, 22:59 and 23:00 boundaries, the rounding and the daylight-saving days are all tested lower down, in unit tests that run in milliseconds; the end-to-end test only proves that the pieces are connected.

Playwright, Selenium and Cypress

Three tools dominate browser automation. All three can run the browser as a headless browser: a real browser engine that loads pages, runs JavaScript and computes the layout, but draws no window on a screen. Headless mode is what pipelines use, since CI (continuous integration) runners have no display; you switch it off (--headed) to watch a test while you debug it.

Playwright is Microsoft's open-source browser automation and testing library. One API drives Chromium, Firefox and WebKit (the engine of Safari), from JavaScript/TypeScript, Python, Java and .NET. Its own test runner, Playwright Test, is for JavaScript and TypeScript; Python users run it through the pytest-playwright plugin.

Selenium WebDriver is the older and most widely supported option: a library for many languages that controls real browsers through the W3C (World Wide Web Consortium) WebDriver standard, with Selenium Grid to run tests on many browsers and machines in parallel. Since version 4.6 the included Selenium Manager downloads the right browser driver for you, so tutorials that start with "download chromedriver" are out of date. Selenium is moving from the classic WebDriver protocol to the newer two-way WebDriver BiDi protocol.

The examples below use Playwright with Python. The ideas (locators, waiting, page objects) apply to all three.

A first Playwright test

With pytest-playwright installed, a test function asks for the page fixture and gets a fresh browser tab, closed again after the test. Relative URLs are resolved against --base-url, so the same test runs against any environment:

import os
import re

from playwright.sync_api import Page, expect


def test_night_reading_is_billed_at_night_rate(page: Page):
    page.goto("/login")
    page.get_by_label("Email address").fill("test.klant06@example.com")
    page.get_by_label("Password").fill(os.environ["PORTAL_TEST_PASSWORD"])
    page.get_by_role("button", name="Log in").click()

    page.get_by_role("link", name="Costs this month").click()
    expect(page.get_by_role("heading", level=1)).to_have_text("Your costs in October 2026")

    row = page.get_by_role("row", name=re.compile("5 October 2026, 23:15"))
    expect(row).to_contain_text("Night")
    expect(row).to_contain_text("€0.34")

pytest tests/e2e -v --browser chromium --base-url https://portal-test.kestrel.example.com

tests/e2e/test_costs.py::test_night_reading_is_billed_at_night_rate[chromium] PASSED [100%]

============================== 1 passed in 4.87s ===============================

Read the test aloud and it is the bug report for BUG-1041 turned into a check: a test customer with a day/night contract, a 1.5 kWh reading at 23:15, and 1.5 × €0.2240 = €0.336, shown as €0.34. The password comes from an environment variable that the pipeline fills from its secret store; it never appears in the test code. The [chromium] after the test name is the browser it ran in: add --browser firefox --browser webkit and the same test runs three times.

Locators: finding elements the way users do

A locator is a test's description of how to find an element on the page: "the button named Log in", "the input labeled Email address". In Playwright a locator is looked up again every time it is used, so it still works after the page has re-rendered. The choice of locator decides how often a test breaks for no good reason:

Playwright's documentation recommends "prioritizing user-facing attributes and explicit contracts such as getByRole". There is a bonus: role and label locators only work if the page exposes roles and accessible names properly. A cost table built from divs instead of a real <table> has no row roles, and the test above cannot find the row; neither can a screen reader. It is not an accessibility test (see Accessibility Testing), but it pushes in the same direction. The CSS selectors and DOM methods behind the fragile options are explained in CSS selectors and finding elements.

Waiting instead of sleeping

A web page is never "ready" at one fixed moment. After a click, the browser sends a request, the portal asks meter-api for readings, the table is filled in by JavaScript. The classic mistake is a fixed pause: time.sleep(2) and then look. On a developer's laptop the table appears in one second and the test passes. On a busy pipeline runner it sometimes takes three, and the test fails although nothing is wrong: a flaky test. Making the sleep longer only makes every run slower.

The fix is to wait for a condition with a time limit. This simulation compares both approaches on 200 runs whose load times vary the way a shared test environment's do (seeded, so the result is the same every time):

import random

random.seed(7)

# How long the cost table took to appear in 200 runs on the shared test
# environment, in milliseconds: usually about one second, sometimes much longer.
load_times = [round(random.lognormvariate(7.0, 0.45)) for _ in range(200)]


def fixed_sleep(load_ms, sleep_ms=2000):
    """time.sleep(2), then look: passes only if the table was already there."""
    return load_ms <= sleep_ms, sleep_ms


def wait_until_visible(load_ms, timeout_ms=5000, poll_ms=100):
    """Check every 100 ms, give up after 5 s (what auto-waiting does)."""
    waited = 0
    while waited < timeout_ms:
        if waited >= load_ms:
            return True, waited
        waited += poll_ms
    return False, waited


for name, strategy in (("sleep(2)", fixed_sleep), ("wait up to 5 s", wait_until_visible)):
    results = [strategy(load) for load in load_times]
    failures = sum(1 for passed, _ in results if not passed)
    waited = sum(ms for _, ms in results) / 1000
    print(f"{name:15} failed {failures:3} of 200 runs, waited {waited:6.1f} s in total")

print(f"slowest load: {max(load_times)} ms, median: {sorted(load_times)[100]} ms")

sleep(2)        failed  15 of 200 runs, waited  400.0 s in total
wait up to 5 s  failed   0 of 200 runs, waited  243.1 s in total
slowest load: 3811 ms, median: 1041 ms

The fixed sleep failed 15 runs out of 200 (7.5%) and was slower: it always waits two seconds, even when the table was there after one. Waiting for the condition never failed and wasted almost no time, because it stops as soon as the table appears.

Auto-waiting in Playwright

Playwright builds this in. Auto-waiting means that before every action the tool waits, up to a time limit, until the element is ready for it. Before click(), Playwright checks that the locator matches exactly one element and that the element is visible, stable (not moving in an animation), able to receive events (not covered by an overlay) and enabled; before fill(), that it is visible, enabled and editable. If the checks do not pass in time, the test fails with a TimeoutError that says which check failed.

Assertions work the same way. expect(row).to_contain_text("€0.34") is a web-first assertion: it keeps checking until the text is there, for up to five seconds by default (expect.set_options(timeout=10_000) changes it). A plain Python assertion such as assert "€0.34" in row.inner_text() reads the text once, immediately, and is flaky for the same reason as the sleep.

In Selenium, waiting is explicit: you wrap the lookup in a WebDriverWait with a condition.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Firefox()  # Selenium Manager finds or downloads the driver
driver.get("https://portal-test.kestrel.example.com/costs")
night_total = WebDriverWait(driver, timeout=5).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, "[data-testid='night-total']"))
)
print(night_total.text)
driver.quit()

Page objects

Once there are a few end-to-end tests, the same steps appear in all of them: find the email field, fill it, click Log in. When the login page changes, every test breaks. The page object model fixes that: each page (or part of a page) gets a class that knows how to find its elements and offers methods in the user's language. Tests call the methods and never touch a locator; when the page changes, only its class changes.

import re

from playwright.sync_api import Page


class LoginPage:
    def __init__(self, page: Page):
        self.page = page

    def open(self):
        self.page.goto("/login")
        return self

    def log_in(self, email: str, password: str) -> "CostPage":
        self.page.get_by_label("Email address").fill(email)
        self.page.get_by_label("Password").fill(password)
        self.page.get_by_role("button", name="Log in").click()
        return CostPage(self.page)


class CostPage:
    def __init__(self, page: Page):
        self.page = page
        self.heading = page.get_by_role("heading", level=1)
        self.night_total = page.get_by_test_id("night-total")

    def open_this_month(self):
        self.page.get_by_role("link", name="Costs this month").click()
        return self

    def row_for(self, moment: str):
        return self.page.get_by_role("row", name=re.compile(moment))

The test now reads like the user story:

import os

from playwright.sync_api import Page, expect

from pages import LoginPage


def test_night_reading_is_billed_at_night_rate(page: Page):
    costs = (
        LoginPage(page).open()
        .log_in("test.klant06@example.com", os.environ["PORTAL_TEST_PASSWORD"])
        .open_this_month()
    )
    expect(costs.row_for("5 October 2026, 23:15")).to_contain_text("€0.34")

Keep assertions in the tests, not in the page objects: a page object answers "how do I do this on this page?", the test decides what is right. Martin Fowler's short essay "PageObject" describes the pattern in more detail.

Visual regression testing

Functional tests check text and behavior; they do not notice that the night column is suddenly white on white or that the chart overlaps the table. Visual regression testing compares screenshots of a page or element with approved reference images (baselines) and fails when they differ by more than a set tolerance. Playwright Test (the JavaScript runner) has it built in:

import { test, expect } from "@playwright/test";

test("September cost chart looks as approved", async ({ page }) => {
  await page.goto("/costs?month=2026-09");
  await expect(page.getByRole("figure", { name: "Costs per day" }))
    .toHaveScreenshot("costs-september.png", { maxDiffPixelRatio: 0.01 });
});

The first run writes the baseline; later runs compare against it, and an intended design change is accepted by updating the baselines (npx playwright test --update-snapshots) in the same pull request, where a reviewer sees the new images. In Python, Playwright's assertions do not include a screenshot comparison; teams take screenshots with page.screenshot() and compare them with a separate plugin or service.

Visual tests are useful and noisy. To keep them honest:

• Fix everything that changes between runs: test data, the date shown on the page (use a fixed month, as above), fonts, the browser version and the operating system that renders the screenshot. A baseline made on a Mac will not match a Linux CI runner.
• Compare components (the chart, the table), not whole pages, so a changed footer does not fail every test.
• Hide or mask dynamic parts such as "last updated at 14:03" and advertisements.

Traces and debugging

When an end-to-end test fails in the pipeline, "element not found" is rarely enough to know why. Was the page still loading, did the login fail, was a cookie banner in the way? Record what happened. Playwright's trace viewer is a tool that replays a recorded test run step by step: for every action it shows the page as it looked before and after (a DOM snapshot you can inspect), the console messages, the network requests and the locator that was used. With pytest-playwright:

# Record a trace, but keep it only when a test fails
pytest tests/e2e --tracing retain-on-failure --screenshot only-on-failure

# Open the trace from the test-results folder (or the pipeline's artifacts)
playwright show-trace test-results/<folder of the failed test>/trace.zip

Let the pipeline upload the test-results/ folder as an artifact, so whoever looks at the red build can open the trace without rerunning anything. Other useful tools: --headed --slowmo 500 to watch a test in slow motion, and playwright codegen, which records your clicks as test code (a starting point to clean up, not a finished test).

End-to-end tests in the pipeline

Some rules that keep end-to-end tests an asset rather than a burden:

• Run them against a dedicated test environment with synthetic test customers (see test data management), never against production with real accounts. A separate, read-only check in production is synthetic monitoring, a different tool with different rules.
• Make each test independent. Every test logs in and creates its own data; no test depends on what an earlier one clicked. Then they can run in parallel and in any order.
• Run them after the fast checks, and on the paths that matter: login, viewing costs, changing a payment method. Unit, integration and contract tests run first and fail faster.
• Treat a flaky end-to-end test as a bug. Record it (the Portal team uses a BUG ticket), find the cause (it is almost always waiting or test data), and quarantine it if it cannot be fixed today. Retrying until green hides real failures; see Flaky Tests, Test Maintenance and Testing in Production.
• Review generated tests like any code. Since version 1.56, Playwright ships test agents that plan, generate and "heal" tests with an AI coding assistant, and recorders such as codegen produce tests in seconds. They save typing; they do not know which journeys matter or what the right price is. A generated test that asserts what the page currently shows will happily confirm a bug.