Practical Patterns: Logs, IP Addresses and Dates

A working method: find, then verify

Most regex work in operations is not validation of user input but searching: which IP addresses appear in this log, which requests failed, when did the errors start. The text is messy and you did not write it. Two kinds of mistake are possible, and a pattern can make both at once.

Overmatching and undermatching

Overmatching is when a pattern matches text it should not: a version number taken for an IP address, or a greedy .* that runs on into the next field. Undermatching is the opposite: the pattern misses text it should find, such as IPv6 addresses, lowercase variants or log lines in a slightly different format. Both are silent: the command runs, prints something plausible, and nobody sees what is wrong.

import re

line = "2026-10-02 09:12:07 WARN login failed user=r.kumar ip=198.51.100.23 tries=3"

print(re.search(r"user=(.*) ", line).group(1))      # greedy: runs on
print(re.search(r"user=(.*?) ", line).group(1))     # lazy: stops early
print(re.search(r"user=([^ ]+)", line).group(1))    # say what a field is
print(re.findall(r"[0-9]+\.[0-9]+", "build 10.4.2 on 198.51.100.23"))

r.kumar ip=198.51.100.23
r.kumar
r.kumar
['10.4', '198.51', '100.23']

The first pattern overmatches: .* is greedy and takes everything up to the last space. The lazy version works here, but [^ ]+ ("one or more characters that are not a space") states what a field is and does not depend on what comes after it. The last line overmatches and undermatches at once: it takes pieces of the version number and of the IP address, and no complete value. See Quantifiers: Greedy, Lazy and Possessive.

The method that keeps both mistakes visible:

1. Write a pattern that finds candidates: loose enough not to miss anything, strict enough not to drown you.
2. Verify each candidate with a real parser, such as Python's ipaddress or datetime, PHP's filter_var or checkdate.
3. Count and inspect what was rejected and which lines produced no match at all.

Pulling fields out of log lines

Log parsing is turning log lines, which are text written for humans, into records with named fields (time, host, severity, user, status) that a program can filter, count and compare. Each field is obtained by field extraction: taking a specific part of a line, usually with a capture group in a pattern or by splitting the line at a separator. Structured logs in JSON need no regex at all; use a JSON parser for them. Regular expressions are for the many logs that are still plain text, such as web server access logs and syslog messages (the Networking topic has a syslog parsing example).

Kestrel Energy's customer portal writes an access log in the common log format, which Apache and nginx can both produce: client address, identity (almost always -), user, time, request, status code and response size.

Build the pattern from one small piece per field, each as precise as the format allows:

import re
from collections import Counter

LOG = """\
192.0.2.15 - amara.okafor [02/Oct/2026:09:12:01 +0000] "POST /login HTTP/1.1" 302 512
198.51.100.23 - - [02/Oct/2026:09:12:07 +0000] "POST /login HTTP/1.1" 401 128
2001:db8::7 - - [02/Oct/2026:09:12:09 +0000] "GET /meters/MTR-100231 HTTP/2.0" 200 2048
-- portal restarted by deploy CHG-4471 --
192.0.2.15 - amara.okafor [02/Oct/2026:09:14:40 +0000] "GET /invoices/2026-09.pdf HTTP/1.1" 500 -
203.0.113.77 - - [02/Oct/2026:09:15:02 +0000] "GET /search?q=meter HTTP/1.1" 200 731
"""

ACCESS = re.compile(
    r'^(?P<client>[^ ]+) [^ ]+ (?P<user>[^ ]+) '
    r'\[(?P<time>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>[^ ]+) HTTP/[0-9.]+" '
    r'(?P<status>[0-9]{3}) (?P<size>[0-9]+|-)$'
)

records, unparsed = [], []
for number, line in enumerate(LOG.splitlines(), start=1):
    m = ACCESS.match(line)
    if m:
        records.append(m.groupdict())
    else:
        unparsed.append((number, line))

print(Counter(r["status"] for r in records).most_common())
for r in records:
    if r["status"] >= "400":
        print(r["status"], r["client"], r["method"], r["path"])
print(f"parsed {len(records)}, not parsed {len(unparsed)}:")
for number, line in unparsed:
    print(f"  line {number}: {line}")

[('200', 2), ('302', 1), ('401', 1), ('500', 1)]
401 198.51.100.23 POST /login
500 192.0.2.15 GET /invoices/2026-09.pdf
parsed 5, not parsed 1:
  line 4: -- portal restarted by deploy CHG-4471 --

groupdict() turns a match into a dictionary with one entry per named group, which is exactly a parsed record. Comparing status codes as strings ("500" >= "400") works here because they always have three digits; convert with int() as soon as you calculate with them. Most important: the line that did not fit is reported, not dropped.

The same with awk

For a quick answer at the prompt, fields separated by spaces are often enough. In the common log format the status code is field 9, so awk can test it directly (see awk):

cat <<'EOF' > access.log
192.0.2.15 - amara.okafor [02/Oct/2026:09:12:01 +0000] "POST /login HTTP/1.1" 302 512
198.51.100.23 - - [02/Oct/2026:09:12:07 +0000] "POST /login HTTP/1.1" 401 128
2001:db8::7 - - [02/Oct/2026:09:12:09 +0000] "GET /meters/MTR-100231 HTTP/2.0" 200 2048
-- portal restarted by deploy CHG-4471 --
192.0.2.15 - amara.okafor [02/Oct/2026:09:14:40 +0000] "GET /invoices/2026-09.pdf HTTP/1.1" 500 -
203.0.113.77 - - [02/Oct/2026:09:15:02 +0000] "GET /search?q=meter HTTP/1.1" 200 731
EOF
echo "--- status codes:"
awk '$9 ~ /^[0-9]{3}$/ { print $9 }' access.log | sort | uniq -c
echo "--- server errors (5xx):"
awk '$9 ~ /^5[0-9][0-9]$/ { print $1, $7 }' access.log

--- status codes:
      2 200
      1 302
      1 401
      1 500
--- server errors (5xx):
192.0.2.15 /invoices/2026-09.pdf

This is quicker to type but more fragile: a malformed request with a space in the path shifts every field after it. The test $9 ~ /^[0-9]{3}$/ at least makes sure that such lines, and the restart message, are not counted as a status code.

Entries that span several lines

Application logs often continue an entry on the next lines, for example with a stack trace. Matching line by line then splits one event into many pieces. A common approach is to treat every line that starts with a timestamp as the start of a new entry and attach all other lines to the previous one:

import re

LOG = """\
2026-10-02 09:14:40 ERROR Invoice export failed for MTR-100231
Traceback (most recent call last):
  File "export.py", line 88, in render
PermissionError: [Errno 13] Permission denied: '/srv/portal/export'
2026-10-02 09:14:41 INFO Retrying export
"""

START = re.compile(r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} [A-Z]+ ", re.ASCII)

entries = []
for line in LOG.splitlines():
    if START.match(line) or not entries:
        entries.append([line])
    else:
        entries[-1].append(line)

for entry in entries:
    print(f"{len(entry)} line(s), starts with: {entry[0][20:]}")

4 line(s), starts with: ERROR Invoice export failed for MTR-100231
1 line(s), starts with: INFO Retrying export

The re.ASCII flag makes \d mean only [0-9] in Python (see \d is not the same everywhere). Many log shippers offer the same idea as a "multiline" setting with a start pattern.

Finding IPv4 addresses

An IPv4 address in dotted-decimal notation is four numbers from 0 to 255 separated by dots. The usual search pattern only describes the shape:

The lookbehind and lookahead in the second pattern say "not preceded and not followed by a digit or a dot", so a match cannot be a piece of a longer number (see Lookahead and Lookbehind). The strict third pattern spells out the numbers 0 to 255 digit by digit. It works, but a parser does the same job with a better error message. Find with the readable pattern, then verify:

import ipaddress
import re

text = ("deny 203.0.113.77 -> 198.51.100.10:443; mask 255.255.255.0; "
        "typo 192.0.2.300; legacy 192.168.01.1; firmware 10.4.2.1.7")

CANDIDATE = re.compile(r"(?<![0-9.])(?:[0-9]{1,3}\.){3}[0-9]{1,3}(?![0-9.])")

for candidate in CANDIDATE.findall(text):
    try:
        address = ipaddress.IPv4Address(candidate)
    except ValueError as error:
        print(f"{candidate:16} rejected: {error}")
    else:
        print(f"{candidate:16} ok")

203.0.113.77     ok
198.51.100.10    ok
255.255.255.0    ok
192.0.2.300      rejected: Octet 300 (> 255) not permitted in '192.0.2.300'
192.168.01.1     rejected: Leading zeros are not permitted in '01' in '192.168.01.1'

Note what happened to 10.4.2.1.7: the boundaries stop the pattern from taking four of its five parts, so a firmware version is not reported as an address. The parser rejects 192.0.2.300 (a part above 255) and 192.168.01.1: Python refuses leading zeros because they are ambiguous: some software reads them as octal numbers, so the same text could mean two different addresses. Also note that 255.255.255.0 is a valid address shape, but here it is a subnet mask; only context tells you which.

PHP has the same check built in as a filter, which also handles IPv6:

<?php
$candidates = ['203.0.113.77', '192.0.2.300', '192.168.01.1', '2001:db8::7', '09:12:07'];
foreach ($candidates as $c) {
    $v4 = filter_var($c, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4) !== false;
    $v6 = filter_var($c, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6) !== false;
    printf("%-14s IPv4: %-3s IPv6: %s\n", $c, $v4 ? 'yes' : 'no', $v6 ? 'yes' : 'no');
}

203.0.113.77   IPv4: yes IPv6: no
192.0.2.300    IPv4: no  IPv6: no
192.168.01.1   IPv4: no  IPv6: no
2001:db8::7    IPv4: no  IPv6: yes
09:12:07       IPv4: no  IPv6: no

IPv6 addresses: let a parser decide

IPv6 notation allows leading zeros to be dropped, one run of zero groups to be replaced by ::, uppercase or lowercase hex digits, an embedded IPv4 address and a zone suffix such as %eth0. A regex that accepts exactly the valid forms exists, but it is so long that nobody can review it. A short, loose candidate pattern plus a parser is far safer. It overmatches on purpose (times and MAC addresses look similar) and leaves the decision to the parser, which also turns every spelling into one canonical form:

import ipaddress
import re

text = ("2026-10-02 09:12:09 login from 2001:db8::7, "
        "relay 2001:0DB8:0000:0000:0000:0000:0000:0001, local ::1, "
        "switch port MAC 00:1a:2b:3c:4d:5e")

CANDIDATE = re.compile(r"(?<![0-9A-Fa-f:])[0-9A-Fa-f]{0,4}(?::[0-9A-Fa-f]{0,4}){2,7}(?![0-9A-Fa-f:])")

for candidate in CANDIDATE.findall(text):
    try:
        address = ipaddress.IPv6Address(candidate)
    except ValueError:
        print(f"{candidate:40} not IPv6")
    else:
        print(f"{candidate:40} {address.compressed}")

09:12:09                                 not IPv6
2001:db8::7                              2001:db8::7
2001:0DB8:0000:0000:0000:0000:0000:0001  2001:db8::1
::1                                      ::1
00:1a:2b:3c:4d:5e                        not IPv6

The time 09:12:09 and the MAC address are found and then rejected; the long uppercase address is recognized and shortened to its canonical form, 2001:db8::1. That normalization matters when you count or compare: the same address can be written in many ways, and only the parsed form tells you that two log lines are about the same host.

Dates and times

Dates are where shape and meaning drift furthest apart. In logs you will meet ISO 8601 (2026-10-02T09:12:07Z), the access log's 02/Oct/2026:09:12:01 +0000, classic syslog's Oct  2 09:12:07 (no year and no time zone), and Dutch-style 02-10-2026 in spreadsheets. Patterns for the common shapes:

Even the strict second pattern only checks that the month is 01–12 and the day 01–31. It cannot know that February has 28 or 29 days, or which years are leap years; a regex that tried would be unreadable. Find with a pattern, then ask a date library:

import re
from datetime import date, datetime

notes = "Meter swap 2026-09-30, billing run 2026-02-30, deadline 2026-13-01"
DATE = re.compile(r"(?<![0-9])[0-9]{4}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12][0-9]|3[01])(?![0-9])")

for found in DATE.findall(notes):
    try:
        print(found, "ok", date.fromisoformat(found).strftime("%A"))
    except ValueError as error:
        print(found, "rejected:", error)

stamp = "2026-10-02T09:12:07+02:00"
print(datetime.fromisoformat(stamp).isoformat())

2026-09-30 ok Wednesday
2026-02-30 rejected: day is out of range for month
2026-10-02T09:12:07+02:00

2026-13-01 was not even a candidate, because the pattern only allows months 01 to 12; 2026-02-30 was, and only the parser caught it. Be careful which parser you trust: some are lenient and quietly roll over an impossible date into the next month. JavaScript's Date does that:

const text = "billing run 2026-02-30";
const m = text.match(/(\d{4})-(\d{2})-(\d{2})/);
const [, year, month, day] = m.map(Number);

const parsed = new Date(Date.UTC(year, month - 1, day));
console.log(parsed.toISOString().slice(0, 10));     // rolled over!

const real = parsed.getUTCFullYear() === year &&
             parsed.getUTCMonth() === month - 1 &&
             parsed.getUTCDate() === day;
console.log("real calendar date:", real);

2026-03-02
real calendar date: false

Comparing the parsed parts with the original ones is the standard check in JavaScript. PHP has a function made for it, checkdate(month, day, year):

<?php
$text = 'Meter swap 30-09-2026, billing run 30-02-2026';
preg_match_all('/\b([0-9]{2})-([0-9]{2})-([0-9]{4})\b/', $text, $dates, PREG_SET_ORDER);

foreach ($dates as [$whole, $day, $month, $year]) {
    $ok = checkdate((int) $month, (int) $day, (int) $year);
    printf("%s -> %s-%s-%s %s\n", $whole, $year, $month, $day, $ok ? 'ok' : 'not a real date');
}

30-09-2026 -> 2026-09-30 ok
30-02-2026 -> 2026-02-30 not a real date

Other everyday identifiers

Many values in IT text have a fixed format that makes them easy to find. A few that turn up at Kestrel Energy, with the pattern a colleague would expect:

Word boundaries (\b) keep an ID from matching inside a longer one. The MAC pattern shows a common undermatching risk: Cisco devices write MAC addresses as three groups of four hex digits, so a NOC search for "all MAC addresses" needs a second alternative. Patterns like these should be tested against the examples in the table before they go into a script:

import re

MAC = re.compile(r"[0-9A-Fa-f]{2}(?:[:-][0-9A-Fa-f]{2}){5}"
                 r"|[0-9A-Fa-f]{4}\.[0-9A-Fa-f]{4}\.[0-9A-Fa-f]{4}")

tests = {
    "00:1a:2b:3c:4d:5e": True,
    "00-1A-2B-3C-4D-5E": True,
    "001a.2b3c.4d5e": True,
    "00:1a:2b:3c:4d": False,
    "09:12:07": False,
    "001a.2b3c": False,
}
for text, expected in tests.items():
    result = MAC.fullmatch(text) is not None
    flag = "ok" if result == expected else "WRONG"
    print(f"{text:18} {str(result):5} {flag}")

00:1a:2b:3c:4d:5e  True  ok
00-1A-2B-3C-4D-5E  True  ok
001a.2b3c.4d5e     True  ok
00:1a:2b:3c:4d     False ok
09:12:07           False ok
001a.2b3c          False ok

A table of strings that must and must not match is the cheapest insurance there is; see Testing, Debugging and Readability. Checking user input, such as email addresses and postal codes, is a different problem with its own risks: Validating Input: Email, Phone Numbers and URLs.

Checking your results

A pattern that runs without errors is not yet a correct pattern. Before you put a number from a log search into a ticket or a report, do three quick checks:

• Do the totals add up? Matching lines plus non-matching lines must equal all lines. If they do, look at the non-matching ones: are any of them lines you meant to count? That is undermatching.
• Do the values make sense? Print the distinct matched values (sort | uniq -c) and look for odd ones out. A status code 999 or an "IP address" 1.2.3.4 in a firmware line is overmatching.
• Did you check case and variants? Try the search again with -i, or with a looser pattern, and compare the counts.

cat <<'EOF' > meter-gateway.log
2026-10-02 08:00:01 MTR-100231 reading 1542.7
2026-10-02 08:00:01 MTR-100232 ERROR timeout
2026-10-02 08:00:02 MTR-100233 reading 877
2026-10-02 08:00:02 MTR-100234 Error: checksum mismatch
2026-10-02 08:00:03 MTR-100235 reading -12.0
2026-10-02 08:00:03 gateway gw-ams-02 ERROR uplink lost
EOF
PATTERN='MTR-[0-9]{6} ERROR'
total=$(wc -l < meter-gateway.log)
matched=$(grep -cE "$PATTERN" meter-gateway.log)
other=$(grep -vcE "$PATTERN" meter-gateway.log)
echo "total $total = matched $matched + other $other"
echo "--- the lines the pattern did not match:"
grep -vE "$PATTERN" meter-gateway.log
echo "--- the same pattern, case-insensitive:"
grep -ciE 'MTR-[0-9]{6} error' meter-gateway.log

total 6 = matched 1 + other 5
--- the lines the pattern did not match:
2026-10-02 08:00:01 MTR-100231 reading 1542.7
2026-10-02 08:00:02 MTR-100233 reading 877
2026-10-02 08:00:02 MTR-100234 Error: checksum mismatch
2026-10-02 08:00:03 MTR-100235 reading -12.0
2026-10-02 08:00:03 gateway gw-ams-02 ERROR uplink lost
--- the same pattern, case-insensitive:
2

Reading the non-matching lines shows two problems: the checksum error is written Error:, so the meter error count of 1 is too low (undermatching); and the negative reading -12.0 is not an error message but probably still worth a ticket. The gateway line correctly does not match, because it is not about a meter. None of this shows up if you only look at the count.