Security Testing

What security testing is

Security testing checks whether software protects its data and functions against misuse: that users can only see and do what they are allowed to, that bad input is rejected safely, that secrets stay secret, and that the parts it is built from have no known holes. Most testing asks "does it do what it should?". Security testing adds the opposite question, "does it refuse everything it should refuse?", which is much harder, because the list of things to refuse is open-ended.

For Kestrel's portal the stakes are concrete: customer names and addresses, contracts, invoices and meter readings, which show when someone is at home. The OWASP Top 10 (2025 edition) is a good map of what usually goes wrong in such an application. Broken access control is still number one, and two categories this page has tests for are newer: software supply chain failures (A03, a wider version of 2021's vulnerable and outdated components) and mishandling of exceptional conditions (A10, new in 2025).

Security testing is not a phase at the end. Like other testing it works best spread over the whole life of the code, with cheap automated checks on every change and expensive human work at the moments that matter. That is the "security as part of normal work" in this page's summary: the Portal team runs most of it themselves, and calls in specialists for what tools cannot do.

The toolbox

The techniques differ in what they look at and when they can run. Three of them are explained on the pipeline testing page and only summarized here: static application security testing (SAST), software composition analysis (SCA) and secret scanning.

You may also meet IAST (interactive application security testing): an agent inside the running application watches how test traffic flows through the code, combining some of the precision of SAST with the realism of DAST. It needs support for your language and runtime, and is less common in small teams.

Scanning code and dependencies

The pipeline page shows how SAST, SCA and secret scanning run on every pull request. Tool names and versions change quickly; as of October 2026:

• Semgrep (1.179.0): pattern-based SAST for many languages, PHP included. The free engine has been called Semgrep Community Edition since December 2024 (previously Semgrep OSS); the rules maintained by Semgrep have their own license, which limits how they may be used.
• CodeQL: GitHub's engine, free for public repositories; private organization repositories need GitHub Code Security.
• Trivy (0.75.0) for images and dependencies, and OWASP Dependency-Check (13.0.0), next to the built-in Dependabot.
• Gitleaks (8.30.1) and TruffleHog (3.99.0) for secrets in history.

Rules for your own mistakes

Generic rule sets catch generic problems. The real strength of a pattern-based tool is a rule for a mistake your team has made. During a test-environment outage, someone once switched off certificate checking in the portal's PHP client for meter-api "for now". The team wrote a Semgrep rule so it cannot happen silently again:

rules:
  - id: kestrel-curl-tls-check-disabled
    languages: [php]
    severity: ERROR
    message: >-
      TLS certificate checks are switched off for this cURL request, so anyone
      on the network path could read or change the data. Remove the option and
      fix the certificate problem instead.
    pattern-either:
      - pattern: curl_setopt($CH, CURLOPT_SSL_VERIFYPEER, false)
      - pattern: curl_setopt($CH, CURLOPT_SSL_VERIFYHOST, 0)

$ semgrep scan --config rules/kestrel.yml --error portal/

┌────────────────┐
│ 1 Code Finding │
└────────────────┘

    portal/lib/meter_client.php
    ❯❯❱ rules.kestrel-curl-tls-check-disabled
          TLS certificate checks are switched off for this cURL request, so anyone
          on the network path could read or change the data. Remove the option and
          fix the certificate problem instead.

           23┆ curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);


Ran 1 rule on 214 files: 1 finding.

--error makes Semgrep exit with a non-zero status when it finds something, which fails the pipeline job.

Triage: the real work

Scanners report candidates, not facts. Every finding needs a decision, and a backlog of 400 untriaged findings is as useless as none. A workable routine:

• Baseline once, then fail on new. When a scanner is introduced, record the existing findings as a backlog with owners, and let the pipeline fail only on findings in changed code. Otherwise the first run blocks every merge for weeks.
• Decide each finding: fix it, accept the risk with a reason and an expiry date, or mark it as a false positive with a note on why. Suppressions in code (// nosemgrep) need the reason next to them.
• Prioritize by reachability and exposure, not by score alone: a critical CVSS score in a library function the portal never calls matters less than a medium one on the login page (see prioritizing beyond the score).
• Run SCA on a schedule too. New vulnerabilities are published for code that has not changed in a year.

Testing the running application with ZAP

Dynamic application security testing (DAST) tests a running application from the outside, the way an attacker would see it: it crawls the pages, sends requests with unusual or malicious input and inspects the responses. It needs no access to the code, so it finds problems of configuration and deployment that SAST cannot see, such as missing security headers or a debug page left switched on.

ZAP (Zed Attack Proxy) is the best-known open-source DAST tool. It was an OWASP project until 2023, and many tutorials still call it "OWASP ZAP"; since September 2024 it is "ZAP by Checkmarx", still open source under the Apache 2.0 license and run by its core team. The latest stable version as of October 2026 is 2.17.0. ZAP works as a proxy between a browser and the application and as a scanner, and ships ready-made scan scripts for pipelines:

• The baseline scan crawls the application for a short time and only looks at the responses (passive scanning). It sends nothing harmful, so it can run on every deployment to the test environment.
• The full scan adds active scanning: it sends attack payloads to every parameter it finds. That can create junk data, lock accounts and overload weak systems. Run it only against a dedicated test environment, with the owners' agreement, never against production.

$ docker run --rm -t -v "$(pwd)":/zap/wrk:rw zaproxy/zap-stable \
    zap-baseline.py -t https://portal-test.kestrel.example.com -r zap-report.html

Total of 31 URLs
PASS: Vulnerable JS Library (Powered by Retire.js) [10003]
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Strict-Transport-Security Header Not Set [10035]
…
WARN-NEW: Content Security Policy (CSP) Header Not Set [10038] x 4 
	https://portal-test.kestrel.example.com/ (200 OK)
	https://portal-test.kestrel.example.com/costs (200 OK)
	https://portal-test.kestrel.example.com/login (200 OK)
	https://portal-test.kestrel.example.com/robots.txt (200 OK)
WARN-NEW: Cookie without SameSite Attribute [10054] x 1 
	https://portal-test.kestrel.example.com/login (200 OK)
WARN-NEW: Server Leaks Version Information via "Server" HTTP Response Header Field [10036] x 31 
	https://portal-test.kestrel.example.com/ (200 OK)
	…
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 3	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 63

Each line names a rule and its ID in brackets. The script's exit status is 1 when there are FAIL findings and 2 when there are only WARN findings (the option -I ignores warnings). A configuration file per application sets each rule to FAIL, WARN or IGNORE, so the scan blocks the pipeline on what the team agreed matters. The findings here are typical for an application that has never been scanned: no Content Security Policy, a session cookie without the SameSite attribute, and a server that announces its version (see security headers). Each becomes a ticket.

Fuzzing

Fuzzing (fuzz testing) feeds a program large amounts of random, malformed or unexpected input and watches for crashes, hangs and broken rules. It started as a University of Wisconsin class project by Barton Miller in 1988, published in 1990, in which random input crashed a large share of standard Unix utilities. It is especially good at finding bugs in code that parses input: Google's OSS-Fuzz service, running since 2016, reported more than 10,000 vulnerabilities found in open-source projects by August 2023.

Kestrel's meter gateway sends lines like M-30007;2026-10-05T23:15;1.500 to a parser. Its contract: return a reading, or raise ReadingError for anything it cannot accept. Any other exception would surface as a "500 Internal Server Error" with a stack trace in the log, exactly the "mishandling of exceptional conditions" of OWASP A10. A simple fuzzer damages a valid line at random and records every other exception:

import random
import re
from datetime import datetime


class ReadingError(ValueError):
    """The only exception the parser is allowed to raise for bad input."""


def parse_reading(line):
    """Parse 'M-30001;2026-10-05T23:15;1.500' from a meter gateway."""
    meter_id, at, kwh = line.split(";")
    if not meter_id.startswith("M-"):
        raise ReadingError(f"bad meter id {meter_id!r}")
    kwh = float(kwh)
    if kwh < 0:
        raise ReadingError("negative reading")
    return meter_id, datetime.fromisoformat(at), kwh


def mutate(text, rng):
    """Damage a valid line a little: delete, insert or replace one character."""
    pos = rng.randrange(len(text))
    action = rng.choice(["delete", "insert", "replace"])
    char = rng.choice(";-.:Te0123456789 x\x00é")
    if action == "delete":
        return text[:pos] + text[pos + 1:]
    if action == "insert":
        return text[:pos] + char + text[pos:]
    return text[:pos] + char + text[pos + 1:]


rng = random.Random(2026)
seed_line = "M-30007;2026-10-05T23:15;1.500"
crashes = {}
for _ in range(2000):
    line = seed_line
    for _ in range(rng.randint(1, 3)):
        line = mutate(line, rng)
    try:
        parse_reading(line)
    except ReadingError:
        pass                                # expected: rejected cleanly
    except Exception as error:              # anything else is a finding
        kind = re.split(r"[:'(]", str(error))[0].strip()  # drop the details
        crashes.setdefault(f"{type(error).__name__}: {kind}", line)

print(f"{len(crashes)} kinds of crash in 2000 inputs")
for key, line in sorted(crashes.items()):
    print(f"{key:52} {line!r}")

8 kinds of crash in 2000 inputs
ValueError: Invalid isoformat string                 'M-30007;20 6-10-05T23:15;1.500'
ValueError: could not convert string to float        'M-30007;2026-10-0\x005T23:15;1.0é'
ValueError: day is out of range for month            'M-30007;2026-10-95T23:15;1.00'
ValueError: hour must be in 0..23                    'M-3000x;2026-10-05T26:15;1.500'
ValueError: minute must be in 0..59                  'M-30007;2026-10-05T239155;1.700'
ValueError: month must be in 1..12                   'M-30007;2026-40-05T23:15;1.500'
ValueError: not enough values to unpack              'M-300072026-10-05T23:15819500'
ValueError: too many values to unpack                'M-30007;2026-10-;523:15;1.500'

Eight kinds of unhandled error in a five-line function, found in a fraction of a second, every one of them caused by input that a real, half-broken gateway could send. The fixed seed makes the run repeatable; a fuzzer in a pipeline uses a new seed each time and logs it, so a failure can be reproduced.

A random fuzzer like this one will never type 1E310, though. Smarter generators can. The test below uses Hypothesis, the property-based testing library, against the improved parser that wraps every parsing error in ReadingError. It generates numbers as text, in every odd form, and checks one property: whatever the parser accepts is a finite, non-negative number of kWh.

import math
from datetime import datetime

from hypothesis import given, seed, strategies as st


class ReadingError(ValueError):
    """The only exception the parser may raise for bad input."""


def parse_reading(line):
    """Parse 'M-30001;2026-10-05T23:15;1.500' from a meter gateway."""
    try:
        meter_id, at, kwh = line.split(";")
        at, kwh = datetime.fromisoformat(at), float(kwh)
    except ValueError as error:
        raise ReadingError(f"unreadable line: {error}") from None
    if not meter_id.startswith("M-") or kwh < 0:
        raise ReadingError("bad meter id or negative reading")
    return meter_id, at, kwh


# Lines that look almost right: a real meter and time, any number as text
numbers = st.one_of(st.floats().map(str), st.text("0123456789.-+eE", max_size=8))
lines = st.builds("M-30007;2026-10-05T23:15;{}".format, numbers)


@seed(2026)
@given(lines)
def test_parser_accepts_only_sane_readings(line):
    try:
        meter_id, at, kwh = parse_reading(line)
    except ReadingError:
        return  # rejected cleanly: fine
    assert math.isfinite(kwh) and kwh >= 0, f"accepted kwh={kwh}"

F                                                                        [100%]
=================================== FAILURES ===================================
____________________ test_parser_accepts_only_sane_readings ____________________

    @seed(2026)
>   @given(lines)
                  

test_example.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

line = 'M-30007;2026-10-05T23:15;1E310'

    @seed(2026)
    @given(lines)
    def test_parser_accepts_only_sane_readings(line):
        try:
            meter_id, at, kwh = parse_reading(line)
        except ReadingError:
            return  # rejected cleanly: fine
>       assert math.isfinite(kwh) and kwh >= 0, f"accepted kwh={kwh}"
E       AssertionError: accepted kwh=inf
E       assert (False)
E        +  where False = <built-in function isfinite>(inf)
E        +    where <built-in function isfinite> = math.isfinite
E       Failing test case: test_parser_accepts_only_sane_readings(
E           line='M-30007;2026-10-05T23:15;1E310',
E       )

test_example.py:35: AssertionError
=========================== short test summary info ============================
FAILED test_example.py::test_parser_accepts_only_sane_readings - AssertionErr...
1 failed in 0.66s

float("1E310") is too big for a float and becomes infinity, which is not negative, so the parser accepted it. An infinite reading would have flowed into the daily total and the invoice. The same hole lets nan through, because every comparison with NaN (not a number) is false. The fix is one line, if not math.isfinite(kwh): raise ReadingError(…), and the test stays in the suite. For long-running, coverage-guided fuzzing of Python code there is Google's Atheris; for C and C++ code, libFuzzer and AFL++ are the usual tools.

Security regression tests

A security regression test is an automated test that reproduces a vulnerability that was found and fixed, and fails if it ever comes back. It is the security version of every regression test, with one difference in emphasis: it checks that something is refused.

During the review of the cost feature, Kwame Asante noticed that meter-api's readings endpoint only checked that a token was valid, not that the meter belonged to the caller: any logged-in customer could read any meter's readings by changing the ID in the URL. That is broken object level authorization, the API form of broken access control. He logged it as BUG-1053 and Priya Shah fixed it. The test that keeps it fixed logs in as one customer and asks for meters that are not theirs:

import json
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import pytest

# A stand-in for the readings API, with fake test data only
TOKENS = {"token-c10001": "C-10001", "token-c10002": "C-10002"}
METERS = {"M-30001": "C-10001", "M-30002": "C-10002"}


class ReadingsAPI(BaseHTTPRequestHandler):
    def do_GET(self):
        customer = TOKENS.get(self.headers.get("Authorization", "").removeprefix("Bearer "))
        if customer is None:
            return self.reply(401, {"error": "not authenticated"})
        meter_id = self.path.removeprefix("/v1/meters/").removesuffix("/readings")
        if METERS.get(meter_id) != customer:  # the fix for BUG-1053
            return self.reply(404, {"error": "not found"})
        self.reply(200, {"meter_id": meter_id, "readings": []})

    def reply(self, status, body):
        data = json.dumps(body).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def log_message(self, format, *args):
        pass


@pytest.fixture(scope="module")
def api():
    server = ThreadingHTTPServer(("127.0.0.1", 0), ReadingsAPI)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    yield f"http://127.0.0.1:{server.server_address[1]}"
    server.shutdown()


def status_of(api, meter_id, token):
    headers = {"Authorization": f"Bearer {token}"} if token is not None else {}
    request = urllib.request.Request(f"{api}/v1/meters/{meter_id}/readings", headers=headers)
    try:
        with urllib.request.urlopen(request) as response:
            return response.status
    except urllib.error.HTTPError as error:
        return error.code


def test_customer_reads_own_meter(api):
    assert status_of(api, "M-30001", "token-c10001") == 200


@pytest.mark.parametrize("meter_id", ["M-30002", "M-30003", "M-99999"])
def test_other_meters_look_like_they_do_not_exist(api, meter_id):
    assert status_of(api, meter_id, "token-c10001") == 404


@pytest.mark.parametrize("token", [None, "token-expired", ""],
                         ids=["missing", "unknown", "empty"])
def test_no_valid_token_no_data(api, token):
    assert status_of(api, "M-30001", token) == 401

============================= test session starts ==============================
collecting ... collected 7 items

test_example.py::test_customer_reads_own_meter PASSED                    [ 14%]
test_example.py::test_other_meters_look_like_they_do_not_exist[M-30002] PASSED [ 28%]
test_example.py::test_other_meters_look_like_they_do_not_exist[M-30003] PASSED [ 42%]
test_example.py::test_other_meters_look_like_they_do_not_exist[M-99999] PASSED [ 57%]
test_example.py::test_no_valid_token_no_data[missing] PASSED             [ 71%]
test_example.py::test_no_valid_token_no_data[unknown] PASSED             [ 85%]
test_example.py::test_no_valid_token_no_data[empty] PASSED               [100%]

============================== 7 passed in 0.68s ===============================

The small server stands in for meter-api so the example runs on its own; in the real suite, the same tests run against the API in the test environment, with two synthetic test customers. Three details make it a good security test:

• It tests the negative cases explicitly: another customer's meter, a meter that belongs to nobody, no token, an unknown token. A functional test suite usually only has the first, positive test.
• It pins the expected refusal: 404 for meters that are not yours, so the API does not even reveal which meter IDs exist, and 401 without a valid token. If someone "simplifies" the check later, the test fails.
• It is named after the rule, not after the bug, so anyone reading a failure understands what broke.

The OWASP testing guide and verification standard

The OWASP Top 10 says what goes wrong most often; it was never meant as a test plan. Two other OWASP documents are:

• The OWASP Web Security Testing Guide (WSTG) is a free handbook of how to test a web application's security, organized in categories (information gathering, configuration, identity, authentication, authorization, session management, input validation and more), each with numbered test cases. The IDOR test above, for example, is WSTG-ATHZ-04 "Testing for Insecure Direct Object References". The stable version is 4.2 (December 2020); version 5 is in development, and the "latest" version on the website is that work in progress.
• The OWASP ASVS (Application Security Verification Standard) is a list of security requirements to verify, in three levels, from a baseline that every application should meet (level 1) to the most critical applications (level 3); level 2 is the usual target for applications that handle sensitive data. Version 5.0.0 was released in May 2025. Because its requirements are testable, a team can use it twice: as security requirements when designing a feature, and as a checklist when testing it.

For the portal, which handles personal data and billing, the Portal team picked ASVS level 2 as its target and turned the relevant requirements into test cases, most of them automated: one test per access rule, one per input field that must be validated, one per cookie flag. The WSTG is what the team reads when it needs to know how to test a requirement.

When to call in a penetration tester

Tools and team tests catch known patterns. A penetration test brings people who think like attackers: they chain three small issues into one big one, abuse business logic ("what if I cancel the contract and request a refund twice?") and find the access rules nobody wrote down. Good moments to commission one:

• Before a major release that changes how customers log in, pay or see data, such as the cost feature, which exposes a customer's consumption per quarter hour.
• After a large architectural change.
• Periodically (often yearly), and when a contract, an auditor or a regulation requires it.

Make the money count. Run the automated scans first and fix what they find, so the testers do not spend expensive days reporting missing headers. Agree the scope and rules of engagement in writing (the security page lists what they contain), prepare a test environment with synthetic data and test accounts for every role, and name a contact who can stop the test if something breaks. Afterward, every finding becomes a ticket with an owner, every fix gets a security regression test, and the testers retest the critical findings.

Ground rules and common mistakes

• "The scan is clean, so we are secure." Scanners find what they know. A generic scanner cannot find the IDOR above, because it does not know that customer A may not see meter M-30002.
• Scanning production "just to check". An active scan can change data and lock accounts. Production gets passive monitoring, scheduled SCA and a disclosure policy.
• Ignoring dependency alerts because the code did not change. The vulnerability is new; the code is old.
• Secrets in test fixtures. Test tokens, API keys and passwords belong in the CI system's secret store, not in the test files, even for the test environment.
• Copies of production data in test environments. A test database with real customers is a data breach waiting to happen; use synthetic data.
• Fixing without a test. A fix without a security regression test is a fix until the next refactoring.

For companies that sell software products in the EU, the Cyber Resilience Act adds a legal reason to take all this seriously: since 11 September 2026, manufacturers must report actively exploited vulnerabilities in their products to the authorities within tight deadlines. Kestrel builds its portal for its own customers rather than selling it, but the suppliers of the products it buys are covered.