Character Classes and Shorthands

Sets in square brackets

A character class matches exactly one character, chosen from a set. The usual way to write one is a bracket expression: the allowed characters between [ and ], where a hyphen between two characters means the whole range from one to the other. [aeiou] is one vowel, [0-9] one digit, [0-9A-F] one uppercase hexadecimal digit. However many characters are listed, the class always stands for one position in the text; add a quantifier to match several.

The last row is a Dutch postal code: four digits that do not start with 0, an optional space, and two capital letters. Ranges follow the order of the characters' code numbers (their code points), so they only make sense between characters of one kind: 0-9, a-z, A-Z.

import re

postal_code = re.compile(r"[1-9][0-9]{3} ?[A-Z]{2}")
for text in ["2511 CV", "1012AB", "0123 AB", "2511 cv"]:
    print(f"{text:8}", bool(postal_code.fullmatch(text)))

2511 CV  True
1012AB   True
0123 AB  False
2511 cv  False

import re

print(re.findall(r"[A-z]+", "KE_0042 ^x[y]"))
print(re.findall(r"[A-Za-z]+", "KE_0042 ^x[y]"))

['KE_', '^x[y]']
['KE', 'x', 'y']

Special characters inside brackets

Inside square brackets the rules change. Most metacharacters lose their meaning: [.] is a dot, [*+?] three literal symbols, [$] a dollar sign. Only four characters are special:

• ] ends the set. To include it, escape it (\]) or, in most flavors, put it first: []a].
• \ escapes, and starts shorthands such as \d (in grep's basic and extended syntax, a backslash inside brackets is just a backslash).
• ^ negates the set, but only as the first character: [^0-9] is "not a digit", [0-9^] is "a digit or a caret".
• - makes a range, except as the first or last character: [-+0-9] and [0-9.-] both contain a literal hyphen.

import re

line = "temp=-4.5C delta=+0.75 setpoint=19"
print(re.findall(r"[-+]?[0-9.]+", line))   # optional sign, digits and dots
print(re.findall(r"[0-9^]+", "2^10 = 1024"))  # caret not first: literal

['-4.5', '+0.75', '19']
['2^10', '1024']

console.log('a]b'.match(/[]a]/g));   // empty class: no match
console.log('a]b'.match(/[\]a]/g));  // escaped: ']' or 'a'
console.log(/[^]/.test('\n'));        // [^] matches anything

null
[ 'a', ']' ]
true

Negated classes

A negated character class starts with ^ and matches one character that is not in the set. [^0-9] is any non-digit, [^"] anything except a double quote. Negated classes are the clearest way to say "up to the next delimiter": "[^"]*" is a double quote, any number of characters that are not quotes, and the closing quote. That stops at the right place, where ".*" would run on to the last quote on the line.

import re

line = 'meter="KE-0042" status="read error" user="amara.okafor"'
print(re.findall(r'"[^"]*"', line))
print(re.findall(r'(\w+)="([^"]*)"', line))

['"KE-0042"', '"read error"', '"amara.okafor"']
[('meter', 'KE-0042'), ('status', 'read error'), ('user', 'amara.okafor')]

One detail surprises people: a negated class does match a newline, because a newline is simply another character that is not in the set. The dot does not match a newline by default, but [^,] does. In multi-line text, add \n to the set if a match must stay on one line:

import re

text = "KE-0042,318.4\nKE-0043,12.0"
print(re.findall(r"[^,]+", text))      # the newline is "not a comma"
print(re.findall(r"[^,\n]+", text))    # stay on one line

['KE-0042', '318.4\nKE-0043', '12.0']
['KE-0042', '318.4', 'KE-0043', '12.0']

Shorthands: \d, \w and \s

A shorthand character class is a backslash and a letter that stands for a commonly used set. The uppercase letter is always the opposite of the lowercase one. Shorthands can also be used inside brackets: [\d.] is a digit or a dot.

"Roughly", because the answer to "which digits? which letters?" depends on the flavor. In plain ASCII text all engines agree, so the differences stay hidden until real-world input arrives: a name with an accent, digits typed on an Arabic keyboard, or a non-breaking space pasted from a spreadsheet.

Unicode or ASCII?

Here is the same pattern pair, \d+ and \w+, on the same text in four tools. The text contains the Western digits 318, the same number written in Arabic-Indic digits (٣١٨), and the word café.

import re

text = "KE-0042 read 318 / ٣١٨ at café"
print(re.findall(r"\d+", text))
print(re.findall(r"\d+", text, re.ASCII))
print(re.findall(r"\w+", text))

['0042', '318', '٣١٨']
['0042', '318']
['KE', '0042', 'read', '318', '٣١٨', 'at', 'café']

const text = 'KE-0042 read 318 / ٣١٨ at café';
console.log(text.match(/\d+/g));
console.log(text.match(/\d+/gu));   // the u flag does not change \d
console.log(text.match(/\w+/g));

[ '0042', '318' ]
[ '0042', '318' ]
[ 'KE', '0042', 'read', '318', 'at', 'caf' ]

<?php
$text = 'KE-0042 read 318 / ٣١٨ at café';
preg_match_all('/\d+/', $text, $plain);
preg_match_all('/\d+/u', $text, $unicode);  // u: treat text as UTF-8
preg_match_all('/\w+/u', $text, $words);
echo implode(' ', $plain[0]), "\n";
echo implode(' ', $unicode[0]), "\n";
echo implode(' ', $words[0]), "\n";

0042 318
0042 318 ٣١٨
KE 0042 read 318 ٣١٨ at café

echo 'KE-0042 read 318 / ٣١٨ at café' > reading.txt
echo "--- grep -oP '\d+':"
grep -oP '\d+' reading.txt
echo "--- grep -oE '[0-9]+':"
grep -oE '[0-9]+' reading.txt

--- grep -oP '\d+':
0042
318
--- grep -oE '[0-9]+':
0042
318

Summed up for these engines on this site's test machine:

Whitespace differs as well. Python's and JavaScript's \s include Unicode spaces such as the non-breaking space (U+00A0) that spreadsheets and web pages often contain; PHP only matches it with the u modifier. If a value "with a space" refuses to split, a non-breaking space is a likely cause.

<?php
$cell = "318.4\u{00A0}kWh";   // value pasted from a spreadsheet
var_dump(preg_match('/\s/', $cell));
var_dump(preg_match('/\s/u', $cell));

int(0)
int(1)

POSIX classes

A POSIX character class is a named set written as [:name:] inside a bracket expression, hence the double brackets in [[:digit:]]. The POSIX standard defines twelve of them. They are the portable way to write character sets for grep, sed and awk, and they follow the locale (the language settings), so [[:alpha:]] includes accented letters in a UTF-8 locale.

Because a POSIX class sits inside a bracket expression, you can combine it with other characters: [[:alnum:]_-] is a letter, digit, underscore or hyphen.

cat <<'EOF' > customers.txt
Jörg Müller     2511 CV
Sanne de Vries  1012AB
Ravi Kumar      3011 kl
EOF
echo "--- words of letters ([[:alpha:]]+):"
grep -oE '^[[:alpha:]]+' customers.txt
echo "--- postal codes with uppercase letters:"
grep -oE '[[:digit:]]{4} ?[[:upper:]]{2}$' customers.txt

--- words of letters ([[:alpha:]]+):
Jörg
Sanne
Ravi
--- postal codes with uppercase letters:
2511 CV
1012AB

PHP (PCRE2) also understands POSIX classes. Python's re and JavaScript do not, and they do not report an error either: they read [[:digit:]] as the set [:digt followed by a literal ]. Python at least prints a FutureWarning ("Possible nested set") on standard error; JavaScript says nothing.

const text = 'meter 4471: ok]';
console.log(text.match(/[[:digit:]]+/g));  // no error, just no match
console.log(text.match(/[0-9]+/g));

null
[ '4471' ]

Unicode property escapes

A Unicode property escape matches one character by its Unicode properties, such as its general category or its script. \p{L} is any letter in any alphabet, \p{Lu} an uppercase letter, \p{Nd} a decimal digit, \p{Script=Greek} a Greek character. The uppercase \P{…} is the opposite. They are the precise way to say "a letter" without listing alphabets, and, unlike \w, they mean essentially the same in every engine that supports them.

const names = ['Jörg', 'Ωmega', 'Lucía', 'R2-D2', 'amara'];
const letters = /^\p{L}+$/u;
const capitalized = /^\p{Lu}\p{Ll}+$/u;
for (const name of names) {
  console.log(name.padEnd(6), letters.test(name), capitalized.test(name));
}

Jörg   true true
Ωmega  true true
Lucía  true true
R2-D2  false false
amara  true false

<?php
foreach (['Jörg', 'Lucía', 'R2-D2'] as $name) {
    echo $name, ': ', preg_match('/^\p{L}+$/u', $name), "\n";
}

Jörg: 1
Lucía: 1
R2-D2: 0

In Python's standard library the same pattern does not compile:

import re

print(re.fullmatch(r"\p{L}+", "Jörg"))

re.error: bad escape \p at position 0

In Python, [^\W\d_] ("a word character that is not a digit or underscore") is a common standard-library substitute for "a letter", because \w is Unicode-aware there. For full property support, use the third-party regex module.

Choosing the right class