Regex on the Command Line: grep, sed and awk — Regular Expressions

TL;DR

grep finds lines, sed rewrites lines, awk splits lines into fields and tests them. All three read text line by line, so they work on files of any size and at any point in a pipe.
Use grep -E (extended syntax) by default; grep -P gives Perl-style extras such as \d and lookaround. Without -P, \d is not a digit: write [0-9].
grep -o prints only the matching part, one per line, which turns grep into an extraction tool: pipe it into sort | uniq -c to count.
sed -E 's/pattern/replacement/g' is the shell's search and replace; & is the whole match and \1 the first group. Try it without -i first.
Always put patterns in single quotes, so the shell does not change *, $ or \ before the tool sees them.

Three tools, three jobs

The classic Unix text tools all take a regular expression, but each one does something different with it:

Tool	What it does with a match	Typical use
`grep`	Prints the line (or only the match)	Find every error for one host in a log
`sed`	Changes the line, then prints it	Mask IP addresses before sharing a log
`awk`	Splits the line into fields and runs a small program	Print the meter ID of every reading that is not a number

The Linux topic introduces grep and its everyday options (-i, -v, -n, -r, -c) and shows how tools are chained with a pipe. This page is about the regular expressions inside those commands: which syntax each tool speaks, and how to extract and rewrite text with them. The examples run with GNU grep 3.8, GNU sed 4.9 and GNU awk (gawk) 5.2, as on most Linux distributions.

Most examples search this log from Kestrel Energy's network operations center (NOC). Each example creates the file itself:

cat <<'EOF' > noc.log
2026-10-02T08:14:03Z rtr-ams-01 %LINK-3-UPDOWN: Interface Gi0/1, changed state to down
2026-10-02T08:14:04Z rtr-ams-01 %LINEPROTO-5-UPDOWN: Line protocol on Interface Gi0/1, changed state to down
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 (192.0.2.41)
2026-10-02T08:17:12Z rtr-ams-01 %LINK-3-UPDOWN: Interface Gi0/1, changed state to up
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
EOF
grep -c 'rtr-ams-01' noc.log
grep 'changed state to up$' noc.log

3
2026-10-02T08:17:12Z rtr-ams-01 %LINK-3-UPDOWN: Interface Gi0/1, changed state to up

grep: basic, extended and Perl-style patterns

GNU grep understands three regex dialects, chosen with an option. The difference is mostly which characters are special without a backslash:

Option	Dialect	Special without backslash	Notes
(none) or `-G`	POSIX basic (BRE)	`. * ^ $ [ ]`	`+ ? \| ( ) { }` are literal; write `\+ \? \\| \{ \}` to make them special
`-E`	POSIX extended (ERE)	`. * ^ $ [ ] + ? \| ( ) { }`	What most people expect a regex to look like
`-P`	Perl-compatible (PCRE2 library)	As ERE, plus `\d \s`, lazy quantifiers, lookaround, `\K`	The same engine family as PHP's preg functions
`-F`	No regex at all	Nothing	Fixed text; fastest and safest for literal strings

grep -E switches grep to POSIX extended regular expressions, in which +, ?, |, parentheses and braces are operators without a backslash. It replaces the old egrep command, which upstream grep 3.8 marks as obsolescent with a warning (Debian-based systems switch that warning off, but keep using -E). Here is the same alternation in each dialect:

cat <<'EOF' > noc.log
2026-10-02T08:14:03Z rtr-ams-01 %LINK-3-UPDOWN: Interface Gi0/1, changed state to down
2026-10-02T08:14:04Z rtr-ams-01 %LINEPROTO-5-UPDOWN: Line protocol on Interface Gi0/1, changed state to down
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 (192.0.2.41)
2026-10-02T08:17:12Z rtr-ams-01 %LINK-3-UPDOWN: Interface Gi0/1, changed state to up
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
EOF
echo "--- ERE: CONFIG|Deny"
grep -E 'CONFIG|Deny' noc.log
echo "--- BRE: CONFIG\|Deny (GNU extension)"
grep 'CONFIG\|Deny' noc.log
echo "--- BRE without backslash: | is a literal character"
grep -c 'CONFIG|Deny' noc.log
echo "exit status: $?"

--- ERE: CONFIG|Deny
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 (192.0.2.41)
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
--- BRE: CONFIG\|Deny (GNU extension)
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 (192.0.2.41)
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
--- BRE without backslash: | is a literal character
0
exit status: 1

The last command found nothing: in basic syntax CONFIG|Deny means the literal text "CONFIG|Deny". grep gives no warning, it simply matches nothing. That silent failure is the strongest argument for typing -E out of habit. The backslashed forms \+, \? and \| in basic syntax are GNU extensions; POSIX leaves their meaning to the implementation, so a script that must also run on BSD or macOS should use -E instead.

The \d trap

\d for "a digit" comes from Perl. Neither basic nor extended POSIX syntax has it. Upstream GNU grep 3.8 warns about such a "stray backslash", but Debian (and therefore Ubuntu and Raspberry Pi OS) switched that warning off, so \d silently means a literal d:

printf '%s\n' 'Interface Gi0/1 down' 'Interface Gid/1 down' > ports.txt
echo "--- -E with \d (matches a literal d):"
grep -E 'Gi\d/1' ports.txt
echo "--- -E with [0-9]:"
grep -E 'Gi[0-9]/1' ports.txt
echo "--- -P with \d:"
grep -P 'Gi\d/1' ports.txt

--- -E with \d (matches a literal d):
Interface Gid/1 down
--- -E with [0-9]:
Interface Gi0/1 down
--- -P with \d:
Interface Gi0/1 down

Write [0-9] (or [[:digit:]]) with -E, or switch to -P when you want Perl shorthands. GNU grep does accept \w, \s, \b, \< and \> in basic and extended syntax as GNU extensions, which makes the missing \d even easier to overlook. The Character Classes and Shorthands page compares these shorthands across engines.

Same pattern, different match

POSIX tools (grep without -P, sed, awk) return the longest match that starts at the leftmost position. Perl-style engines, including grep -P, try the alternatives in order and stop at the first one that works. With alternatives that are prefixes of each other, the result differs:

echo 'Escalated to INC-31902' > note.txt
echo "--- grep -oE (POSIX, longest):"
grep -oE 'INC|INC-[0-9]+' note.txt
echo "--- grep -oP (Perl-style, first alternative that matches):"
grep -oP 'INC|INC-[0-9]+' note.txt

--- grep -oE (POSIX, longest):
INC-31902
--- grep -oP (Perl-style, first alternative that matches):
INC

Put the longer alternative first, or write INC(-[0-9]+)?, and every engine agrees. The Alternation and Precedence page explains both rules.

Perl-style extras with grep -P

-P uses the PCRE2 library. Two features make it worth knowing on the command line: lookbehind, and \K, which "forgets" everything matched so far. Both let grep -o print a value without the label in front of it:

cat <<'EOF' > noc.log
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 (192.0.2.41)
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
EOF
echo "--- \K: the user name after 'by '"
grep -oP 'by \K[a-z]+' noc.log
echo "--- lookbehind: the address after 'src outside:'"
grep -oP '(?<=src outside:)[0-9.]+' noc.log

--- \K: the user name after 'by '
tomasz
--- lookbehind: the address after 'src outside:'
203.0.113.77

The grep manual still calls -P experimental in combination with -z, and grep must have been built with PCRE2 support, as the Debian package used here is; otherwise -P reports that it is not supported. The PCRE2 version comes from the system library, so features added in newer PCRE2 releases, such as bounded variable-length lookbehind in 10.43, depend on your distribution. See Lookahead and Lookbehind.

Printing only the match: grep -o

By default grep prints whole lines. grep -o prints only the parts of each line that match, each on a line of its own, so one line with three matches gives three output lines. That changes grep from a filter into an extraction tool. Combined with sort and uniq -c, it answers "how often does each value occur?" in one line:

cat <<'EOF' > shift-notes.txt
08:10 Portal slow for customers in Utrecht, see INC-31902
08:25 INC-31902 linked to CHG-4471 (load balancer change)
09:02 New ticket INC-31917: meter MTR-100232 offline
09:40 INC-31902 and INC-31917 both waiting for CHG-4471 back-out
EOF
echo "--- every incident ID, as often as it appears:"
grep -oE 'INC-[0-9]{5}' shift-notes.txt
echo "--- counted:"
grep -oE 'INC-[0-9]{5}' shift-notes.txt | sort | uniq -c
echo "--- lines with an incident ID (-c counts lines, not matches):"
grep -cE 'INC-[0-9]{5}' shift-notes.txt
echo "--- matches (-o, then count the output lines):"
grep -oE 'INC-[0-9]{5}' shift-notes.txt | wc -l

--- every incident ID, as often as it appears:
INC-31902
INC-31902
INC-31917
INC-31902
INC-31917
--- counted:
      3 INC-31902
      2 INC-31917
--- lines with an incident ID (-c counts lines, not matches):
4
--- matches (-o, then count the output lines):
5

Note the difference between the last two numbers: -c counts lines that contain at least one match, and the last line holds two IDs. To count matches, count the output of -o. Also note that the pattern says exactly five digits; a pattern such as INC-[0-9]+ would also pick up a mistyped INC-319020, which may or may not be what you want.

grep in scripts: exit status and -q

grep's exit status is 0 when at least one line matched, 1 when nothing matched and 2 on an error, such as a missing file or an invalid pattern. With -q (quiet) it prints nothing and only sets the status, which is exactly what an if needs:

printf '%s\n' 'Port 8443' 'TLSProtocol TLSv1.2 TLSv1.3' > portal.conf
if grep -qE '^TLSProtocol .*TLSv1\.[01]\b' portal.conf; then
  echo "WARN: old TLS version enabled"
else
  echo "OK: no TLS 1.0 or 1.1"
fi
grep -E '(' portal.conf
echo "exit status: $?"

OK: no TLS 1.0 or 1.1
grep: Unmatched ( or \(
exit status: 2

Status 2 matters in monitoring scripts: a check written as "alert if grep finds nothing" will also stay silent if the pattern is broken or the log file is missing, unless the script tells 1 and 2 apart.

sed: editing a stream

sed (stream editor) reads text line by line, applies editing commands to each line and writes the result to standard output. It does not open an editor window and, by default, does not change the input file. Most sed commands start with an optional address that selects lines, often a regular expression between slashes, followed by a one-letter command: /^#/d deletes comment lines, /ERROR/p prints error lines. By far the most used command is s.

The s command

A sed substitution, written s/pattern/replacement/flags, replaces the first match of the pattern on each line with the replacement text; the g flag replaces every match on the line. In the replacement, & stands for the whole match and \1 to \9 for capture groups. sed uses basic syntax by default; -E switches it to extended syntax, just like grep.

cat <<'EOF' > noc.log
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 (192.0.2.41)
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
EOF
IP='([0-9]{1,3}\.){3}[0-9]{1,3}'
echo "--- without g: only the first address per line"
sed -E "s/$IP/[ip]/" noc.log
echo "--- with g: every address"
sed -E "s/$IP/[ip]/g" noc.log
echo "--- & is the whole match"
echo 'Disk usage at 81%' | sed -E 's/[0-9]+%/(&)/'

--- without g: only the first address per line
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 ([ip])
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:[ip]/51544 dst inside:198.51.100.10/443
--- with g: every address
2026-10-02T08:15:40Z sw-rtd-02 %SYS-5-CONFIG_I: Configured from console by tomasz on vty0 ([ip])
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:[ip]/51544 dst inside:[ip]/443
--- & is the whole match
Disk usage at (81%)

The pattern is kept in a shell variable here, so the double-quoted sed command must be read carefully: inside double quotes the shell replaces $IP but leaves the backslashes in this pattern alone. Without a variable, single quotes are the safer choice. Masking addresses like this is a reasonable first step before you paste a log into a ticket or an AI assistant, but check the result: a pattern for IPv4 addresses does nothing for IPv6 addresses, host names or user names.

Groups in the replacement

Capture groups let you rearrange a match. A common chore is turning Dutch-style dates (day-month-year) into ISO 8601 order, which sorts correctly as plain text:

cat <<'EOF' > readings.csv
02-10-2026;MTR-100231;1542.7
02-10-2026;MTR-100232;ERR
30-09-2026;MTR-100231;1529.4
EOF
sed -E 's/^([0-9]{2})-([0-9]{2})-([0-9]{4});/\3-\2-\1;/' readings.csv
echo "--- the same in basic syntax (more backslashes):"
sed 's/^\([0-9]\{2\}\)-\([0-9]\{2\}\)-\([0-9]\{4\}\);/\3-\2-\1;/' readings.csv | head -n 1

2026-10-02;MTR-100231;1542.7
2026-10-02;MTR-100232;ERR
2026-09-30;MTR-100231;1529.4
--- the same in basic syntax (more backslashes):
2026-10-02;MTR-100231;1542.7

The extended version is far easier to read. The anchor ^ and the ; at the end make sure only the date field at the start of a line is touched, never a number later in the line that happens to look like a date.

Other delimiters, -n and flags

The slash after s is only a convention: any character can be the delimiter, which avoids escaping every slash in a path. -n turns off the automatic printing of every line, so with the p flag sed prints only lines where a substitution happened. GNU sed also has an I flag for case-insensitive matching.

cat <<'EOF' > portal.log
2026-10-02 09:12:01 INFO login ok user=amara.okafor ip=192.0.2.15
2026-10-02 09:12:07 WARN login failed user=r.kumar ip=198.51.100.23
2026-10-02 09:12:30 INFO export path=/srv/portal/export/2026-10.csv
EOF
echo "--- a different delimiter for paths:"
sed 's|/srv/portal/export/|/data/exports/|' portal.log | tail -n 1
echo "--- -n with p: print only the extracted user names"
sed -nE 's/.* user=([a-z.]+) .*/\1/p' portal.log
echo "--- I flag: case-insensitive"
sed -n 's/warn/[!]/Ip' portal.log

--- a different delimiter for paths:
2026-10-02 09:12:30 INFO export path=/data/exports/2026-10.csv
--- -n with p: print only the extracted user names
amara.okafor
r.kumar
--- I flag: case-insensitive
2026-10-02 09:12:07 [!] login failed user=r.kumar ip=198.51.100.23

Changing files in place, safely

sed -i writes the result back into the file instead of to the screen. That is how configuration changes are scripted, and it is also how a wrong pattern destroys a file in a fraction of a second. A safe routine: run the command without -i and read the output, then use -i.bak so sed keeps a backup, and compare:

printf '%s\n' '# Kestrel portal' 'listen 8080' 'upstream_timeout 30' > portal.conf
sed -i.bak -E 's/^listen [0-9]+$/listen 8443/' portal.conf
diff portal.conf.bak portal.conf
echo "diff exit status: $?"

2c2
< listen 8080
---
> listen 8443
diff exit status: 1

diff exits with status 1 when the files differ, which is the expected result here. The anchored pattern ^listen [0-9]+$ changes only a line that is exactly "listen" and a port number, not a comment that mentions listening. BSD sed, as on macOS, always expects a suffix after -i (sed -i '' … for no backup), and GNU sed does not accept that separate empty argument. -i.bak written as one word works in both.

awk: patterns on fields

awk is a small programming language for text that splits each input line into fields and runs pattern { action } rules on it: whenever the pattern is true for a line, the action runs. The fields are $1, $2 and so on, $0 is the whole line and NF the number of fields. By default fields are separated by runs of spaces and tabs; -F sets another separator. awk's regular expressions use extended syntax, and can be applied to the whole line (/ERROR/) or to one field ($4 ~ /^[0-9]+$/, or !~ for "does not match").

awk splits each line into fields first; a regex test on $4 looks at that field alone.

Testing one field is the big advantage over grep: grep '^[0-9]' cannot easily say "the fourth column is a number", awk can. Here it separates good meter readings from bad ones:

cat <<'EOF' > meters.log
2026-10-02 08:00 MTR-100231 1542.7 kWh
2026-10-02 08:00 MTR-100232 ERR timeout
2026-10-02 08:00 MTR-100233 877 kWh
2026-10-02 08:00 MTR-100234 -12.0 kWh
EOF
echo "--- valid readings:"
awk '$4 ~ /^[0-9]+(\.[0-9]+)?$/ { print $3, $4 }' meters.log
echo "--- everything else:"
awk '$4 !~ /^[0-9]+(\.[0-9]+)?$/ { print $3 ": " $4, $5 }' meters.log
echo "--- whole-line pattern, then a field:"
awk '/ERR/ { print "check meter", $3 }' meters.log

--- valid readings:
MTR-100231 1542.7
MTR-100233 877
--- everything else:
MTR-100232: ERR timeout
MTR-100234: -12.0 kWh
--- whole-line pattern, then a field:
check meter MTR-100232

The anchors matter: without ^ and $, [0-9]+ would also accept -12.0 and ERR1, because each contains digits.

Regex functions in awk

awk has functions that take a regex: sub and gsub replace the first or every match in a field or line (with & for the match, as in sed), match finds a match and sets RSTART and RLENGTH, its position and length, and split cuts a string at a regex. gawk adds gensub, which supports group references such as \\1 (two backslashes inside an awk string).

cat <<'EOF' > tickets.csv
INC-31902;P2;Portal slow for customers in Utrecht
INC-31917;P3;Meter MTR-100232 offline since 08:00
CHG-4471;-;Load balancer pool change
EOF
echo "--- -F';' and a field test: incidents only"
awk -F';' '$1 ~ /^INC-/ { print $1, $2 }' tickets.csv
echo "--- gsub: mask meter IDs in the description"
awk -F';' '{ gsub(/MTR-[0-9]+/, "MTR-xxxxxx", $3); print $3 }' tickets.csv
echo "--- match, RSTART and RLENGTH: pull out a time"
awk 'match($0, /[0-9]{2}:[0-9]{2}/) { print substr($0, RSTART, RLENGTH) }' tickets.csv
echo "--- gensub (gawk only): swap number and prefix"
awk -F';' '{ print gensub(/^([A-Z]+)-([0-9]+)$/, "\\2 (\\1)", 1, $1) }' tickets.csv

--- -F';' and a field test: incidents only
INC-31902 P2
INC-31917 P3
--- gsub: mask meter IDs in the description
Portal slow for customers in Utrecht
Meter MTR-xxxxxx offline since 08:00
Load balancer pool change
--- match, RSTART and RLENGTH: pull out a time
08:00
--- gensub (gawk only): swap number and prefix
31902 (INC)
31917 (INC)
4471 (CHG)

Like grep, awk has no \d. gawk at least says so, on standard error, and then treats the escape as a literal d, so the line does not match:

echo 'MTR-100231 1542' | gawk '/\d+/ { print "matched" }'
echo "--- with [0-9]:"
echo 'MTR-100231 1542' | gawk '/[0-9]+/ { print "matched" }'

gawk: cmd. line:1: warning: regexp escape sequence `\d' is not a known regexp operator
--- with [0-9]:
matched

Putting the tools together

Each tool does one step well, so real questions are answered with short pipelines. Which outside addresses did the firewall deny most often this morning?

cat <<'EOF' > fw.log
2026-10-02T08:21:55Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51544 dst inside:198.51.100.10/443
2026-10-02T08:22:01Z fw-ams-01 %ASA-6-302013: Built inbound TCP connection for outside:192.0.2.80/50110 to inside:198.51.100.10/443
2026-10-02T08:22:09Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51561 dst inside:198.51.100.10/22
2026-10-02T08:23:30Z fw-ams-01 %ASA-4-106023: Deny udp src outside:198.51.100.200/5353 dst inside:198.51.100.10/161
2026-10-02T08:24:12Z fw-ams-01 %ASA-4-106023: Deny tcp src outside:203.0.113.77/51602 dst inside:198.51.100.10/3389
EOF
grep -F '%ASA-4-106023' fw.log \
  | grep -oE 'src outside:[0-9.]+' \
  | sed 's/^src outside://' \
  | sort | uniq -c | sort -rn
echo "--- the same in one awk program:"
awk '/%ASA-4-106023/ { split($7, a, "[:/]"); n[a[2]]++ }
     END { for (ip in n) print n[ip], ip }' fw.log | sort -rn

      3 203.0.113.77
      1 198.51.100.200
--- the same in one awk program:
3 203.0.113.77
1 198.51.100.200

Both answers agree. The first pipeline is easier to build step by step: run it after each | and look at what comes out. The awk version reads the file once, which matters for gigabytes of logs. The final sort is needed in the awk version because the order of a for (ip in n) loop is not defined. For more patterns of this kind, see Practical Patterns: Logs, IP Addresses and Dates.

ripgrep

ripgrep (command rg) is a newer, open-source alternative to grep -r for searching directory trees, popular with developers. By default it searches recursively, skips files listed in .gitignore, hidden files and binary files, and uses Rust's regex engine, which runs in linear time but has no lookaround and no backreferences; -P switches to PCRE2 when you need those. It is not installed by default on most servers, so learn grep first (illustrative, not run):

$ rg -n 'INC-[0-9]{5}' docs/          # extended-style syntax, recursive by default
$ rg -P 'by \K[a-z]+' noc.log         # PCRE2 for \K and lookaround

Quoting, locales and portability

Three things decide whether a command-line regex does what you typed:

Shell quoting. The shell reads the command line before grep, sed or awk do. Unquoted, * and ? can be replaced by file names, $ starts a variable, and the shell removes backslashes. Single quotes pass the pattern through untouched; use double quotes only when you deliberately insert a shell variable.
Locale. Ranges and classes such as [a-z] and [[:alpha:]] follow the LC_ALL/LANG settings. In a UTF-8 locale [[:alpha:]] matches letters such as "é" and "ü"; in the C locale it matches only ASCII letters. For predictable results in scripts, prefer explicit sets like [0-9] and [A-Za-z], or set the locale.
GNU versus the rest. \+, \? and \| in basic syntax, \w and \b, sed's I flag, sed -i without a suffix argument and awk's gensub are GNU behavior. On BSD and macOS tools, they may fail or mean something else.

printf '%s\n' 'Eindhoven' 'Zürich' 'Den Haag' > cities.txt
echo "--- single-word city names, [[:alpha:]] in a UTF-8 locale:"
grep -xE '[[:alpha:]]+' cities.txt
echo "--- the same with [A-Za-z]:"
grep -xE '[A-Za-z]+' cities.txt

--- single-word city names, [[:alpha:]] in a UTF-8 locale:
Eindhoven
Zürich
--- the same with [A-Za-z]:
Eindhoven

Neither answer is wrong; they answer different questions. Decide whether "letter" means ASCII or any letter, and write the class that says so. The Regex Flavors page puts the differences between POSIX tools and programming languages side by side.