Lookahead and Lookbehind

What lookaround is

Sometimes the text you want is defined by its surroundings: the number before "kWh", the amount after "EUR", the hostname that does not start with "test-". You could match the surroundings too and pick out a capture group, and often that is the simplest way. But sometimes you want the surroundings checked and left out of the match itself, for example in a replacement or with grep -o.

Lookaround is the collective name for assertions that check whether a sub-pattern matches just after or just before the current position, without consuming the text it looks at. Like ^ and \b, a lookaround is a zero-width assertion: it succeeds or fails, and the match position stays where it was. There are four kinds:

Lookahead

A lookahead, written (?=…), succeeds if the text right after the current position matches the pattern inside it; the text it checks stays available to the rest of the pattern and is not part of the match. Typical use: "a number, but only when a unit follows".

import re

readings = "KE-0042: 318 kWh, KM-0107: 92 kWh, read 7 days late"
print(re.findall(r"\d+ kWh", readings))
print(re.findall(r"\d+(?= kWh)", readings))

['318 kWh', '92 kWh']
['318', '92']

The first pattern includes the unit in the result; the lookahead version returns only the numbers. The 7 and the digits in the meter IDs are skipped, because no kWh follows them. A capture group, (\d+) kWh, would give the same list with findall; the lookahead becomes really useful in replacements and in tools that print the whole match, such as grep -o.

Several conditions at one position

Because a lookahead does not move the position, you can stack several at the start of a pattern, and each checks the same text from the beginning. That is how "must contain a digit and a capital letter" is written as one regex: ^(?=.*\d)(?=.*[A-Z]).{12,}$. It is the textbook example, but think twice before you enforce such composition rules on passwords: length helps far more than a required digit (see good password practice). A cleaner use is checking a hostname label: letters, digits and hyphens, at most 63 characters, not starting or ending with a hyphen.

import re

label = re.compile(r"(?!-)[a-z0-9-]{1,63}(?<!-)")
for name in ["noc-sw02", "-noc", "noc-", "portal", "n" * 64]:
    print(f"{name[:12]:13} {label.fullmatch(name) is not None}")

noc-sw02      True
-noc          False
noc-          False
portal        True
nnnnnnnnnnnn  False

(?!-) at the start rejects a leading hyphen, (?<!-) at the end rejects a trailing one, and the character class in between does the rest. For whole domain names and URLs, a parser is still the better tool; see Validating Input.

Negative lookahead

A negative lookahead, written (?!…), succeeds only if the text right after the current position does not match the pattern inside it. It is the regex way to say "but not": hostnames that are not test machines, lines that do not contain a word.

import re

hosts = ["portal.example.com", "test-portal.example.com", "api.example.com", "test-api.example.com"]
print([h for h in hosts if re.fullmatch(r"(?!test-)[a-z-]+\.example\.com", h)])
print(re.findall(r"\b(?!test-)[a-z-]+\.example\.com", " ".join(hosts)))

log = """08:14 INFO link up
08:15 DEBUG poll ok
08:16 ERROR fan failed"""
print(re.findall(r"^(?!.*DEBUG).*$", log, re.MULTILINE))

['portal.example.com', 'api.example.com']
['portal.example.com', '-portal.example.com', 'api.example.com', '-api.example.com']
['08:14 INFO link up', '08:16 ERROR fan failed']

The first line works because fullmatch tests each hostname from its first character. The second line searches one long string, and two results come out wrong: the engine skips the excluded test and starts a match at the hyphen, where \b also succeeds. A negative lookahead only guards the position where it stands, so make sure that position is the one you mean.

The last pattern reads: at the start of a line, make sure that "anything followed by DEBUG" cannot be found from here, then take the whole line. In a shell, grep -v DEBUG does the same job more simply, but inside a program or a larger pattern the lookahead is handy.

The backtracking trap

A negative lookahead only checks the exact position where the engine happens to be. If the part before it can match less, the engine backtracks until it finds a position where the lookahead succeeds:

import re

readings = "318 kWh, 92 kWh, 7 days"
print(re.findall(r"\d+(?! kWh)", readings))
print(re.findall(r"\b\d+\b(?! kWh)", readings))

['31', '9', '7']
['7']

The intention was "numbers that are not followed by kWh", yet 31 and 9 come out: after 31 the next text is 8 kWh, not kWh, so the lookahead is satisfied. Adding \b after \d+ forces the number to be complete before the lookahead is tested. Whenever you write a negative lookahead after a quantifier, ask what happens if the quantifier gives back one character.

Lookbehind

A lookbehind, written (?<=…), succeeds if the text right before the current position matches the pattern inside it; its negative form (?<!…) succeeds if it does not. Use it for "the value after a label" or "a number that is not part of a ticket ID".

import re

invoice = "Total EUR 112.50, discount EUR 12.00, paid USD 99.99"
print(re.findall(r"(?<=EUR )\d+\.\d{2}", invoice))

note = "CHG-1042 window 0915-1030, room 2511"
print(re.findall(r"(?<!CHG-)\b\d{4}\b", note))

['112.50', '12.00']
['0915', '1030', '2511']

const invoice = "Total EUR 112.50, discount EUR 12.00, paid USD 99.99";
console.log(invoice.match(/(?<=EUR )\d+\.\d{2}/g));
console.log(invoice.replace(/(?<=EUR )\d+\.\d{2}/g, "***"));

[ '112.50', '12.00' ]
Total EUR ***, discount EUR ***, paid USD 99.99

The replacement shows why lookbehind is handy: the label stays, only the amount is replaced, without having to put the label back with a group reference.

Lookbehind limits per engine

Lookahead works the same everywhere it exists. Lookbehind does not: looking backwards is harder for an engine that reads left to right, so many engines require the lookbehind to have a fixed length, which lets them step back exactly that many characters and test from there. Here is one variable-length lookbehind, "digits preceded by one or more capitals and a hyphen", in three languages:

console.log("INC-31877 CHG-1042".match(/(?<=[A-Z]+-)\d+/g));

[ '31877', '1042' ]

import re

print(re.findall(r"(?<=INC-|CHG-)\d+", "INC-31877 CHG-1042"))
print(re.findall(r"(?<=[A-Z]+-)\d+", "INC-31877 CHG-1042"))

['31877', '1042']
re.error: look-behind requires fixed-width pattern

<?php
var_dump(preg_match_all('/(?<=INC-|CHG-)\d+/', "INC-31877 CHG-1042"));
var_dump(preg_match_all('/(?<=[A-Z]+-)\d+/', "INC-31877 CHG-1042"));

int(2)
bool(false)

JavaScript accepts the variable-length lookbehind. Python and PHP compile the version with two fixed alternatives of equal length, but reject [A-Z]+: Python raises an error, PHP prints a warning ("lookbehind assertion is not fixed length") and returns false, which is not the same as 0 matches. Always compare preg_* results with ===.

Lookaround in grep, and how to do without it

In the shell, only grep -P (Perl-compatible syntax, via PCRE2) supports lookaround. It also offers \K, which means "forget what was matched so far": everything before it must match but is left out of the result, like a lookbehind without the length limit.

printf '%s\n' 'meter=KE-0042 kwh=318' 'meter=KM-0107 kwh=92' 'meter=KE-0311 status=offline' > readings.txt
grep -o -P '(?<=meter=)[A-Z]{2}-\d{4}(?= kwh=)' readings.txt
echo ---
grep -o -P 'kwh=\K\d+' readings.txt
echo ---
grep -E 'kwh=' readings.txt | grep -v 'KM-'

KE-0042
KM-0107
---
318
92
---
meter=KE-0042 kwh=318

The last command shows the general fallback when there is no lookaround: split the job into simple steps, one that selects and one that excludes. In code, the usual alternative is a capture group: match the surroundings as well, and read only the group you need (kwh=(\d+), then group 1). That works in every engine, including RE2, and is often easier to read than a lookaround.