Testing, Debugging and Readability

Build a pattern step by step

Regular expressions are dense: one wrong character changes the meaning, and a long pattern that "doesn't work" gives no hint about where it goes wrong. So write them the way you would debug them: start with a small piece that surely matches, run it against real lines, and add one part at a time.

Say the NOC wants the meter ID and the reading from smart-meter gateway logs. Each step below is run against three real lines, including a broken one:

import re

lines = [
    "2026-10-02T08:14:03Z gw03 READING meter=KE-0042 kwh=318.4 status=OK",
    "2026-10-02T08:14:05Z gw03 READING meter=KE-0107 kwh=12 status=OK",
    "2026-10-02T08:14:09Z gw07 READING meter=KE-00421 kwh=-1 status=ERR",
]

steps = [
    r"meter=",
    r"meter=KE-[0-9]{4}",
    r"meter=KE-[0-9]{4}\b",
    r"meter=(KE-[0-9]{4})\b kwh=([0-9]+(?:\.[0-9]+)?)\b",
]

for step in steps:
    print(step)
    for line in lines:
        m = re.search(step, line)
        print("   ", m.groups() if m and m.groups() else (m.group() if m else None))

meter=
    meter=
    meter=
    meter=
meter=KE-[0-9]{4}
    meter=KE-0042
    meter=KE-0107
    meter=KE-0042
meter=KE-[0-9]{4}\b
    meter=KE-0042
    meter=KE-0107
    None
meter=(KE-[0-9]{4})\b kwh=([0-9]+(?:\.[0-9]+)?)\b
    ('KE-0042', '318.4')
    ('KE-0107', '12')
    None

Step 2 reveals a bug that a finished pattern would have hidden: the third line has a malformed ID, KE-00421, and the pattern happily reports it as KE-0042, a different meter. That is overmatching in its most dangerous form: a wrong answer that looks right. The word boundary \b in step 3 fixes it. Without the step-by-step run, nobody would have noticed until a customer got the wrong bill.

Test cases

A regex test case is an input string paired with the result the pattern should give for it: match or no match, and, where it matters, which text or groups it should capture. A list of test cases turns "I think it works" into something you can run again after every change, like any other automated test. A library can also generate inputs for you; see Property-Based Testing.

Positive and negative examples

Good test lists have two halves. Positive and negative examples are the inputs a pattern must match (positive) and the inputs it must reject (negative). Most people write only the positive half, and a pattern like .* passes every positive test. The negative half is where the bugs are. Good negative examples are near misses: one character too many or too few, the wrong case, a trailing space, an empty string, digits from another script.

Here is a check for a Kestrel Energy incident number field, tested against both halves:

import re

SHOULD_MATCH = ["INC-31902", "INC-00017"]
SHOULD_NOT_MATCH = ["INC-3190", "INC-319021", "inc-31902", "CHG-1234",
                    "INC-31902 ", "INC-٣١٩٠٢", ""]

def run_tests(name, is_ticket):
    failures = [v for v in SHOULD_MATCH if not is_ticket(v)]
    failures += [v for v in SHOULD_NOT_MATCH if is_ticket(v)]
    total = len(SHOULD_MATCH) + len(SHOULD_NOT_MATCH)
    print(f"{name}: {total - len(failures)}/{total} passed", failures)

run_tests("first try", lambda v: re.search(r"INC-\d{5}", v) is not None)
run_tests("fixed    ", lambda v: re.fullmatch(r"INC-[0-9]{5}", v) is not None)

first try: 6/9 passed ['INC-319021', 'INC-31902 ', 'INC-٣١٩٠٢']
fixed    : 9/9 passed []

The first try passed both positive tests and would have looked fine in a quick manual check. Three negative tests caught two separate bugs: no anchoring (so INC-319021 and the trailing space slip through) and \d accepting Arabic-Indic digits in Python. The fix uses re.fullmatch() and [0-9].

In a real project, put these lists in your test suite so they run on every change; see unit tests in Python. With unittest, subTest reports each failing input separately:

import re
import unittest

TICKET = re.compile(r"INC-[0-9]{5}")


class TicketPatternTest(unittest.TestCase):
    def test_accepts_valid_ids(self):
        for value in ["INC-31902", "INC-00017"]:
            with self.subTest(value=value):
                self.assertIsNotNone(TICKET.fullmatch(value))

    def test_rejects_near_misses(self):
        for value in ["INC-3190", "INC-319021", "inc-31902", "INC-31902\n", ""]:
            with self.subTest(value=value):
                self.assertIsNone(TICKET.fullmatch(value))


if __name__ == "__main__":
    unittest.main()

When a bug report comes in ("ticket INC-31902 was not linked"), first add the input as a test case, watch it fail, then fix the pattern. The test then stays as a regression test.

Test in the flavor you deploy

A pattern is only tested for the engine you ran it in. The same text ^INC-\d{5}$ in three languages, with the same three inputs:

import re

for value in ["INC-31902", "INC-31902\n", "INC-٣١٩٠٢"]:
    print(repr(value), re.search(r"^INC-\d{5}$", value) is not None)

'INC-31902' True
'INC-31902\n' True
'INC-٣١٩٠٢' True

for (const value of ["INC-31902", "INC-31902\n", "INC-٣١٩٠٢"]) {
  console.log(JSON.stringify(value), /^INC-\d{5}$/.test(value));
}

"INC-31902" true
"INC-31902\n" false
"INC-٣١٩٠٢" false

<?php
foreach (["INC-31902", "INC-31902\n", "INC-٣١٩٠٢"] as $value) {
    echo json_encode($value, JSON_UNESCAPED_UNICODE), ' ',
        preg_match('/^INC-\d{5}$/', $value), ' ',
        preg_match('/^INC-\d{5}$/u', $value), "\n";
}

"INC-31902" 1 1
"INC-31902\n" 1 1
"INC-٣١٩٠٢" 0 1

Python accepts all three. JavaScript accepts only the first: its $ does not match before a final newline and its \d is ASCII-only. PHP accepts the newline, and accepts the Arabic-Indic digits only with the u modifier. A pattern copied from a JavaScript front end to a Python back end therefore needs its tests run again in Python. The differences are collected on regex flavors.

Debugging a pattern

See exactly what matched

"It matches" is not enough information. Print the matched text and its position, every time, until the pattern behaves. The helper below marks each match under the text, which makes an overmatching .* obvious at a glance:

import re

def show(pattern, text):
    """Print the text with ^ under every character the pattern matched."""
    print(pattern)
    print("  " + text)
    marks = [" "] * len(text)
    for m in re.finditer(pattern, text):
        start, end = m.span()
        marks[start:end] = "^" * (end - start)
    print("  " + "".join(marks).rstrip())

line = 'login user="amara" host="portal01" result="ok"'
show(r'"(.*)"', line)
show(r'"([^"]*)"', line)

"(.*)"
  login user="amara" host="portal01" result="ok"
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
"([^"]*)"
  login user="amara" host="portal01" result="ok"
             ^^^^^^^      ^^^^^^^^^^        ^^^^

The greedy .* runs from the first quote to the last one (see quantifiers); [^"]* cannot cross a quote, so it stops at the right place. On the command line, grep -o does the same job: it prints only the matched part of each line.

Make invisible characters visible

When a pattern that "obviously" fits does not match, the input often contains something you cannot see: a trailing space, a tab, a non-breaking space, or the carriage return (CR) that Windows tools put before every line feed (LF). GNU sed's l command prints lines with such characters spelled out:

# An export from a Windows tool: lines end in CR LF
printf 'INC-31902\r\nINC-31903\r\nINC-31904\n' > tickets.txt

grep -cE '^INC-[0-9]{5}$' tickets.txt     # expected 3

# Make the invisible visible: sed's l command shows \r and ends lines with $
sed -n 'l' tickets.txt

# Fix the input, not the pattern
tr -d '\r' < tickets.txt | grep -cE '^INC-[0-9]{5}$'

1
INC-31902\r$
INC-31903\r$
INC-31904$
3

cat -A and Python's repr() show the same information. In Python, re.compile(pattern, re.DEBUG) goes one step further and prints how the engine parsed the pattern. The exact format differs between Python versions, but it can settle an argument quickly; in this excerpt from Python 3.11, UNI_DIGIT shows that \d means "any Unicode digit":

SUBPATTERN 1 0 0
  MAX_REPEAT 1 MAXREPEAT
    IN
      CATEGORY CATEGORY_DIGIT
…
13.     CATEGORY UNI_DIGIT

Common bugs and their fixes

The raw-string row is explained on Python strings and literals, metacharacters and escaping.

Readable patterns

A pattern is read far more often than it is written, usually by someone in a hurry during an incident. A one-line pattern of 80 characters is hard to review and easy to break. Three tools help: verbose mode, named groups and building a pattern from parts.

Verbose mode and comments

Verbose mode (Python's re.VERBOSE, PCRE's x modifier) ignores whitespace in the pattern, so you can spread it over several lines and indent it. It also allows comments. A pattern comment is text inside a regular expression that the engine ignores: in verbose mode everything from # to the end of the line, and in Python and PCRE also an inline (?#…) group that works without verbose mode. Combined with named groups, the meter-reading pattern becomes self-documenting:

import re

READING = re.compile(r"""
    ^(?P<time> [0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]{8}Z ) \s+   # ISO 8601, UTC
    (?P<gateway> gw[0-9]{2} ) \s+                        # gateway name
    READING \s+
    meter=(?P<meter> KE-[0-9]{4} ) \b \s+                # 4 digits, no more
    kwh=(?P<kwh> [0-9]+ (?:\.[0-9]+)? ) \b               # 12 or 318.4
""", re.VERBOSE)

line = "2026-10-02T08:14:03Z gw03 READING meter=KE-0042 kwh=318.4 status=OK"
m = READING.search(line)
print(m.groupdict())

{'time': '2026-10-02T08:14:03Z', 'gateway': 'gw03', 'meter': 'KE-0042', 'kwh': '318.4'}

In verbose mode a literal space or # must be escaped (\ , \#) or written as a class ([ ]), because unescaped ones are ignored. PHP does the same with the x modifier. Note that PHP names groups with (?<name>…), a form Python 3.11 rejects:

<?php
$reading = '/
    meter=(?<meter> KE-[0-9]{4} ) \b \s+   # meter ID, exactly 4 digits
    kwh=(?<kwh> [0-9]+ (?:\.[0-9]+)? ) \b  # whole or decimal number
/x';

$line = '2026-10-02T08:14:03Z gw03 READING meter=KE-0042 kwh=318.4 status=OK';
preg_match($reading, $line, $m);
echo $m['meter'], ' ', $m['kwh'], "\n";

// (?#...) is an inline comment, without the x modifier
echo preg_match('/INC-(?#incident)[0-9]{5}/', 'INC-31902'), "\n";

KE-0042 318.4
1

Building a pattern from named parts

JavaScript has no verbose flag and no (?#…) comments (both are errors in Node.js 24). The usual alternative is to build the pattern from small strings with ordinary code comments and join them. String.raw keeps the backslashes as typed:

// JavaScript has no x flag: build the pattern from named, commented parts.
const parts = [
  String.raw`meter=(?<meter>KE-[0-9]{4})\b`, // meter ID, exactly 4 digits
  String.raw`\s+`,
  String.raw`kwh=(?<kwh>[0-9]+(?:\.[0-9]+)?)\b`, // whole or decimal number
];
const reading = new RegExp(parts.join(""));

const line = "2026-10-02T08:14:03Z gw03 READING meter=KE-0042 kwh=318.4 status=OK";
console.log(reading.source);
console.log(line.match(reading).groups);

meter=(?<meter>KE-[0-9]{4})\b\s+kwh=(?<kwh>[0-9]+(?:\.[0-9]+)?)\b
[Object: null prototype] { meter: 'KE-0042', kwh: '318.4' }

The same idea works in every language: give reusable pieces a name (METER_ID, DECIMAL) and compose them. Only insert fixed pieces this way; text from users must be escaped first, or it becomes part of the pattern.

Online regex testers

A regex tester is a tool, usually a web page, where you type a pattern and sample text and see the matches, groups and an explanation of each part update as you type. They are excellent for learning and for the step-by-step building shown above. Two widely used ones:

• regex101.com lets you choose the flavor. As of 2026 it offers PCRE2 (PHP), JavaScript, Python, Golang, Java, .NET, Rust, POSIX ERE and BRE, and legacy PCRE. It explains every token in plain language.
• RegExr (regexr.com) focuses on the JavaScript and PCRE flavors and has a reference panel and community patterns.

Two cautions. First, always select the flavor that matches where the pattern will run, and still run your tests in the real language: escaping inside a string literal, flags and API differences (re.match() versus re.search()) are invisible in a tester. Second, whatever you paste into a website leaves your control. Use invented sample lines, never production logs with customer data, email addresses, tokens or passwords; for real data, test locally with grep or a short script.

This topic has its own tester: the Regex Tester runs in your browser with the JavaScript engine, sends nothing anywhere and stops patterns that run too long.

Checking AI-written patterns

AI assistants write regular expressions quickly and often well. They also produce patterns for the wrong flavor, with missing anchors, or with nested quantifiers, delivered with the same confidence as correct ones. Treat a pattern from an AI assistant exactly like one from a forum or a colleague's old script: as code from a stranger that you test before you trust it (see checking AI output). A short checklist:

1. Flavor: was it written for the engine you use? (?<name>…) fails in Python 3.11, \d means something else in grep, and possessive quantifiers fail in JavaScript.
2. Positive and negative examples: run it on your own test list, especially near misses, empty input, very long input and non-ASCII text.
3. Anchoring: does it check the whole value when it should?
4. Backtracking risk: look for nested or overlapping quantifiers and test with a long failing input (see performance).
5. Understanding: can you explain every part? If not, ask for the verbose version with comments, or simplify it. You will be the one debugging it at 3 a.m.

Asking the assistant to also generate test cases is useful, but add your own negative examples: a model that misunderstood the requirement will write tests that share the misunderstanding.

Keeping patterns maintainable

• Keep the examples next to the pattern. A comment with one matching and one non-matching example tells the next reader more than a description.
• Give compiled patterns names (TICKET_ID, METER_READING) and define them once, not copied into five places.
• Run the tests automatically on every change, so a "small improvement" to a pattern cannot silently break a case someone fixed last year.
• Prefer the simpler pattern when two are equivalent. Clever patterns save a line and cost an afternoon.
• Document the flavor when a pattern is stored outside code (a config file, a monitoring rule, a firewall rule): the next system that reads it may use a different engine.