Text Processing in Scripts

The sample log

Every example on this page works on the same small log from Kestrel Energy's meter-reading service. Each line has a date, a time, a level, the word meter, a meter ID and a message:

2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout

The commands themselves are explained in the Linux topic (grep, filters) and the Regular Expressions topic (grep, sed and awk). This page is about using them in scripts, next to their PowerShell counterparts.

Searching for lines

In Bash, grep prints the lines that match a pattern; -c counts them, -n adds line numbers and -i ignores case:

cat > app.log <<'EOF'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
EOF
grep -n ERROR app.log
echo "Errors: $(grep -c ERROR app.log)"

2:2026-10-06 09:00:07 ERROR meter M-1042 timeout
4:2026-10-06 09:02:30 ERROR meter M-1044 timeout
Errors: 2

Select-String is PowerShell's equivalent of grep: it searches files or pipeline strings with a regular expression and returns a MatchInfo object per matching line. Displayed, such an object looks like grep -n output (file, line number, line); in a script you use its properties Line, LineNumber and Matches. Unlike grep, it ignores case unless you add -CaseSensitive:

@'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
'@ | Set-Content app.log

$errors = Select-String -Path app.log -Pattern 'ERROR'
$errors | ForEach-Object { "$($_.LineNumber): $($_.Line)" }
"Errors: $($errors.Count)"
"Lines with 'error' in any case: $((Select-String -Path app.log -Pattern 'error').Count)"
"Case-sensitive 'error': $((Select-String -Path app.log -Pattern 'error' -CaseSensitive).Count)"

2: 2026-10-06 09:00:07 ERROR meter M-1042 timeout
4: 2026-10-06 09:02:30 ERROR meter M-1044 timeout
Errors: 2
Lines with 'error' in any case: 2
Case-sensitive 'error': 0

"Did it match?" in a condition

Scripts often only need a yes or no. grep -q prints nothing and reports through its exit status: 0 when a line matched, 1 when none did, 2 on an error such as a missing file. Select-String -Quiet returns $true when something matched. When nothing matched, it returns $false for files given with -Path, but nothing at all ($null) for piped strings. Test it with if or -not, never with -eq $false:

cat > app.log <<'EOF'
2026-10-06 09:00:07 ERROR meter M-1042 timeout
EOF
if grep -q 'ERROR' app.log; then
  echo "Errors found: open a ticket"
fi
if ! grep -q 'CRITICAL' app.log; then
  echo "No critical lines"
fi

Errors found: open a ticket
No critical lines

'2026-10-06 09:00:07 ERROR meter M-1042 timeout' | Set-Content app.log
if (Select-String -Path app.log -Pattern 'ERROR' -Quiet) {
    'Errors found: open a ticket'
}
if (-not (Select-String -Path app.log -Pattern 'CRITICAL' -Quiet)) {
    'No critical lines'
}

Errors found: open a ticket
No critical lines

Extracting fields

A field separator is the character or pattern that divides a line into fields: a comma in CSV, a tab in TSV (tab-separated values), runs of spaces in most logs. Every field-based tool needs to know it. awk splits on runs of spaces and tabs by default and takes another separator with -F; cut uses a single character given with -d; read uses the characters in the shell variable IFS.

cat > app.log <<'EOF'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
EOF
echo "Meters with errors (awk):"
awk '$3 == "ERROR" { print "  " $5 }' app.log
echo "Levels (cut):"
cut -d' ' -f3 app.log | sort -u | sed 's/^/  /'
echo "Time and level (read):"
while read -r date time level _; do
  echo "  $time $level"
done < app.log

Meters with errors (awk):
  M-1042
  M-1044
Levels (cut):
  ERROR
  INFO
  WARN
Time and level (read):
  09:00:01 INFO
  09:00:07 ERROR
  09:01:12 WARN
  09:02:30 ERROR

The _ at the end of read is a throwaway variable: read puts all remaining fields into its last variable, so naming one you ignore keeps level clean. cut treats every single space as a separator, so a line with two spaces in a row has an empty field and the numbering shifts; awk does not have that problem with its default separator.

In PowerShell, split the line into an array with -split (see splitting and joining) and pick elements by index. A unary -split, without a pattern, splits on runs of whitespace just like awk:

@'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
'@ | Set-Content app.log

'Meters with errors:'
foreach ($line in Get-Content app.log) {
    $fields = -split $line
    if ($fields[2] -eq 'ERROR') { "  $($fields[4])" }
}
'Levels:'
Get-Content app.log | ForEach-Object { (-split $_)[2] } | Sort-Object -Unique |
    ForEach-Object { "  $_" }

Meters with errors:
  M-1042
  M-1044
Levels:
  ERROR
  INFO
  WARN

Arrays in PowerShell count from 0, so field 3 in awk ($3) is $fields[2]. Get-Content returns the file as an array of lines, one string each; with -Raw it returns the whole file as one string, which is what you want for replacements that span lines.

Matching and capturing with regular expressions

To pull a value out of the middle of a line, match it with a regular expression and capture the part you need. Bash offers three ways: grep -o prints only the matching part, sed -n 's/…/\1/p' prints a capture group, and the test [[ $line =~ regex ]] stores capture groups in the array BASH_REMATCH:

cat > app.log <<'EOF'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
EOF
echo "grep -o:"
grep -o 'M-[0-9]\{4\}' app.log | sort | uniq -c
echo "sed capture:"
sed -n 's/.*ERROR meter \(M-[0-9]*\).*/  \1/p' app.log
echo "[[ =~ ]]:"
ticket='INC-20419'
if [[ $ticket =~ ^INC-([0-9]{5})$ ]]; then
  echo "  ticket number ${BASH_REMATCH[1]}"
fi

grep -o:
      2 M-1042
      1 M-1043
      1 M-1044
sed capture:
  M-1042
  M-1044
[[ =~ ]]:
  ticket number 20419

Note the two dialects: grep and sed without -E use basic regular expressions, where a repetition count is written \{4\} and a group \( \); [[ =~ ]] uses extended ones, where they are {5} and ( ). Keep the pattern for =~ unquoted (or in a variable): quoting it makes Bash match it as a literal string. POSIX BRE and ERE lists the differences.

The match operator -match is PowerShell's regular-expression test: it returns $true or $false, and when the left side is a single string it fills the automatic variable $Matches, a hash table with the whole match under key 0, numbered groups under 1, 2 and so on, and named groups under their names. PowerShell uses the .NET regular-expression engine, which supports \d, named groups (?<name>…) and lookarounds (see .NET and Java):

$ticket = 'INC-20419'
if ($ticket -match '^INC-(\d{5})$') {
    "Whole match: $($Matches[0])"
    "Group 1:     $($Matches[1])"
}

$line = '2026-10-06 09:00:07 ERROR meter M-1042 timeout'
if ($line -match '(?<level>[A-Z]+) meter (?<meter>M-\d{4})') {
    "Level: $($Matches.level), meter: $($Matches.meter)"
}

Whole match: INC-20419
Group 1:     20419
Level: ERROR, meter: M-1042

With an array on the left, -match behaves differently: it returns the elements that match and does not set $Matches. That makes it a compact filter. To collect every capture in a file, read the groups from Select-String results instead:

@'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
'@ | Set-Content app.log
$lines = Get-Content app.log

'Filter with -match on an array:'
$lines -match 'timeout' | ForEach-Object { "  $_" }
'Captures through Select-String:'
Select-String -Path app.log -Pattern 'ERROR meter (M-\d{4})' |
    ForEach-Object { '  ' + $_.Matches[0].Groups[1].Value }

Filter with -match on an array:
  2026-10-06 09:00:07 ERROR meter M-1042 timeout
  2026-10-06 09:02:30 ERROR meter M-1044 timeout
Captures through Select-String:
  M-1042
  M-1044

Search and replace

sed replaces text in a stream with s/pattern/replacement/; add g to replace every match on a line, -E for extended regular expressions, and -i to edit a file in place (see sed substitution). Here meter IDs are masked in a log copy that will be attached to a supplier ticket:

cat > app.log <<'EOF'
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:02:30 ERROR meter M-1044 timeout
EOF
sed -E 's/M-[0-9]{4}/M-xxxx/g' app.log
echo "---"
sed -E 's/^([0-9-]+) ([0-9:]+)/\2 on \1/' app.log

2026-10-06 09:00:07 ERROR meter M-xxxx timeout
2026-10-06 09:02:30 ERROR meter M-xxxx timeout
---
09:00:07 on 2026-10-06 ERROR meter M-1042 timeout
09:02:30 on 2026-10-06 ERROR meter M-1044 timeout

The replace operator -replace is PowerShell's equivalent: <input> -replace <regex>, <replacement> replaces every match and returns the new string (or array of strings). In the replacement, $1 refers to group 1 and ${name} to a named group, which is why the replacement belongs in single quotes:

$line = '2026-10-06 09:00:07 ERROR meter M-1042 timeout'
$line -replace 'M-\d{4}', 'M-xxxx'
$line -replace '^([\d-]+) ([\d:]+)', '$2 on $1'
'With double quotes:'
$line -replace '^([\d-]+) ([\d:]+)', "$2 on $1"

2026-10-06 09:00:07 ERROR meter M-xxxx timeout
09:00:07 on 2026-10-06 ERROR meter M-1042 timeout
With double quotes:
 on  ERROR meter M-1042 timeout

In double quotes PowerShell expands $2 and $1 as (empty) PowerShell variables before -replace ever sees them, so the date and time vanish. The same applies to the Bash side the other way round: keep sed scripts in single quotes so the shell leaves \1 and $ alone.

Because the pattern is a regular expression, characters such as ., \ and ( have a special meaning. To replace a literal text, escape it with [regex]::Escape(), or use the string method .Replace(), which does no regular expressions at all (and is case-sensitive):

$path = 'C:\Logs\kestrel\app.log'
$path -replace [regex]::Escape('C:\Logs'), 'D:\Archive'
$path.Replace('C:\Logs', 'D:\Archive')
'1.5.10' -replace '.', '-'
'1.5.10' -replace '\.', '-'

D:\Archive\kestrel\app.log
D:\Archive\kestrel\app.log
------
1-5-10

To change a file, read it, replace and write it back. Wrap Get-Content in parentheses so the whole file is read before Set-Content opens it for writing; Bash's sed -i takes care of this itself by writing to a temporary file:

"server=192.0.2.10`nport=8443" | Set-Content app.conf
(Get-Content app.conf) -replace '^server=.*', 'server=192.0.2.20' | Set-Content app.conf
Get-Content app.conf

server=192.0.2.20
port=8443

printf 'server=192.0.2.10\nport=8443\n' > app.conf
sed -i 's/^server=.*/server=192.0.2.20/' app.conf
cat app.conf

server=192.0.2.20
port=8443

Splitting and joining

The split operator -split divides a string into an array of substrings at every match of a regular expression; -join does the reverse. The regular expression part is a common trap: a dot means "any character", so splitting a version number on '.' gives only empty strings:

$version = '7.6.6'
"Split on '.':  $(($version -split '.').Count) empty pieces"
"Split on '\.': $(($version -split '\.') -join ' | ')"
"SimpleMatch:   $(($version -split '.', 0, 'SimpleMatch') -join ' | ')"

$csvLine = 'M-1042,Utrecht,,4521'
$fields = $csvLine -split ','
"Fields: $($fields.Count), third is empty: $($fields[2] -eq '')"
"Joined: $($fields -join ';')"

Split on '.':  6 empty pieces
Split on '\.': 7 | 6 | 6
SimpleMatch:   7 | 6 | 6
Fields: 4, third is empty: True
Joined: M-1042;Utrecht;;4521

In Bash, read -ra splits a string into an array using IFS as the separator, and tr or parameter expansion (${list//,/;}) changes separators. Setting IFS on the same line as read changes it for that command only:

version='5.2.15'
IFS=. read -ra parts <<< "$version"
echo "Parts: ${#parts[@]}, major ${parts[0]}, minor ${parts[1]}"

csv_line='M-1042,Utrecht,,4521'
IFS=, read -ra fields <<< "$csv_line"
echo "Fields: ${#fields[@]}, third is empty: [${fields[2]}]"
echo "Joined: ${csv_line//,/;}"

Parts: 3, major 5, minor 2
Fields: 4, third is empty: []
Joined: M-1042;Utrecht;;4521

<<< is a here-string: it feeds the value of a variable to a command's standard input. Neither -split ',' nor IFS=, understands quoted CSV fields such as "Utrecht, Centrum"; for real CSV use Import-Csv or a CSV-aware tool (see JSON, CSV and APIs).

Counting and summarizing

The classic Bash summary is a pipeline: cut out the field, sort, count duplicates with uniq -c, and sort the counts numerically. PowerShell's Group-Object does the counting in one step on any value you compute:

cat > app.log <<'EOF'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
EOF
awk '{ print $3 }' app.log | sort | uniq -c | sort -k1,1nr -k2

      2 ERROR
      1 INFO
      1 WARN

@'
2026-10-06 09:00:01 INFO meter M-1042 read ok
2026-10-06 09:00:07 ERROR meter M-1042 timeout
2026-10-06 09:01:12 WARN meter M-1043 slow response
2026-10-06 09:02:30 ERROR meter M-1044 timeout
'@ | Set-Content app.log

Get-Content app.log |
    Group-Object { (-split $_)[2] } |
    Sort-Object @{ Expression = 'Count'; Descending = $true }, Name |
    ForEach-Object { '{0,7} {1}' -f $_.Count, $_.Name }

      2 ERROR
      1 INFO
      1 WARN

Both versions sort by count first and by name second, so equal counts always come out in the same order. Without that second key, the order of ties is not guaranteed, and a report that changes order from run to run makes comparisons between days needlessly hard.

Case sensitivity

The two shells have opposite defaults, and that difference causes real bugs when a script is translated from one to the other:

$name = 'Kestrel Energy'
"-like '*ENERGY*':       $($name -like '*ENERGY*')"
"-clike '*ENERGY*':      $($name -clike '*ENERGY*')"
".Contains('ENERGY'):    $($name.Contains('ENERGY'))"
"'ERROR' -eq 'error':    $('ERROR' -eq 'error')"

-like '*ENERGY*':       True
-clike '*ENERGY*':      False
.Contains('ENERGY'):    False
'ERROR' -eq 'error':    True

Side by side

When text tools are not enough

Text tools are at their best on line-oriented data: logs, configuration files, command output. They become fragile when the data has structure that spans lines or hides separators inside values: CSV with quoted commas, JSON, XML, HTML. A grep on JSON breaks the day the producer pretty-prints its output differently. Use a parser for structured formats (jq, Import-Csv, ConvertFrom-Json; see JSON, CSV and APIs), and consider Python once an awk program needs more than a few lines. The Pipeline: Text versus Objects explains why PowerShell scripts usually need less of this parsing to begin with.