Loops

Loops in scripts

Loops are where scripts start to pay off: the same check on twenty servers, the same conversion for every CSV file of the night, the same test until a service answers. The general idea of a loop is explained in Control Structures › Iteration; the Bash basics are in Shell Scripting Basics › Loops. This page compares the two shells and concentrates on the traps that cost admins time: lists that split in the wrong places, lines that disappear, and counters that stay at zero.

for in Bash

The for loop in Bash runs its body once for every word in a list, with the loop variable set to that word. The list is whatever the shell produces after expansion: words you type, an array, a glob or the output of a command. That last step is where it goes wrong, because unquoted values are split at spaces:

servers=(web01 db01 "backup server")

for s in "${servers[@]}"; do
  echo "checking: $s"
done
echo "--- without the quotes:"
for s in ${servers[@]}; do
  echo "checking: $s"
done

checking: web01
checking: db01
checking: backup server
--- without the quotes:
checking: web01
checking: db01
checking: backup
checking: server

"${servers[@]}" in double quotes gives each array element as exactly one word, spaces and all. Without the quotes, "backup server" becomes two servers. Arrays themselves are explained in Arrays, Hash Tables and Objects.

Counting

For a fixed range, brace expansion ({1..3}) is short, but it happens before variables are expanded, so {1..$n} does not work. Use the C-style form with double parentheses whenever a number comes from a variable:

n=3
for i in {1..3}; do echo "brace expansion: $i"; done
for i in {1..$n}; do echo "brace with a variable: $i"; done
for (( i = 1; i <= n; i++ )); do echo "C-style: $i"; done

brace expansion: 1
brace expansion: 2
brace expansion: 3
brace with a variable: {1..3}
C-style: 1
C-style: 2
C-style: 3

Files: globs, not ls

Loop over files with a glob (see wildcards), which hands every file name over as one word. The output of ls is text, and $(ls) splits it at every space:

touch "meter 10.log" "meter 9.log" app.log

for f in $(ls); do echo "[$f]"; done
echo "--- glob:"
for f in *.log; do echo "[$f]"; done

[app.log]
[meter]
[10.log]
[meter]
[9.log]
--- glob:
[app.log]
[meter 10.log]
[meter 9.log]

ShellCheck reports the first form as SC2045 ("Iterating over ls output is fragile. Use globs."), and for f in $(find …) as SC2044. A glob that matches nothing stays as the literal pattern unless nullglob is set; for recursive searches, feed find -print0 into a while read loop. Both are covered in Files, Paths and Globbing.

foreach and ForEach-Object in PowerShell

PowerShell has two ways to do something for every item, and their names are confusingly similar.

The foreach statement

The foreach statement, foreach ($item in $collection) { … }, is part of the language. It first evaluates the whole collection, then runs the block once per item with the loop variable you named. Because it does not go through the pipeline, it is the fastest loop for data that is already in memory, and break and continue work as expected.

$servers = 'web01', 'db01', 'backup server'
foreach ($s in $servers) {
    "checking: $s"
}

$n = 3
foreach ($i in 1..$n) { "range: $i" }
for ($i = 1; $i -le $n; $i++) { "for: $i" }

checking: web01
checking: db01
checking: backup server
range: 1
range: 2
range: 3
for: 1
for: 2
for: 3

PowerShell variables are not split at spaces, so "backup server" stays one item without any quoting rules. The range operator 1..$n accepts variables, unlike Bash's {1..3}. The C-style for exists too, but a range reads better.

The ForEach-Object cmdlet

ForEach-Object is a cmdlet that runs a script block for every object that arrives through the PowerShell pipeline, with the current object in $_ (or $PSItem). It processes each object as soon as it arrives, so it can handle a 2 GB log without loading it all, and it fits in the middle of a pipeline. Optional -Begin and -End blocks run once before the first and after the last object:

'M-1042', 'M-1043', 'M-1044' |
    ForEach-Object -Begin { 'export started' } -Process { "exporting meter $_" } -End { 'export done' }

'web01', 'db01' | ForEach-Object ToUpper

export started
exporting meter M-1042
exporting meter M-1043
exporting meter M-1044
export done
WEB01
DB01

The last line uses the short form: give ForEach-Object the name of a property or method, and it reads or calls it on every object. In interactive use you will see the aliases % and foreach for ForEach-Object; that second alias is why foreach at the start of a statement and foreach after a | are two different things. In scripts, write the full cmdlet name.

Collecting results

Whatever a loop body outputs and does not capture is output of the loop, so you can assign the loop itself to a variable. That is simpler and much faster than starting with an empty array and adding to it with +=, which copies the whole array on every round:

$costs = foreach ($kwh in 4521, 312, 9870) {
    [math]::Round($kwh * 0.28, 2)
}
"collected $($costs.Count) values: $($costs -join ', ')"

collected 3 values: 1265.88, 87.36, 2763.6

ForEach-Object -Parallel

Since PowerShell 7.0, ForEach-Object -Parallel runs the script block for several objects at the same time (five by default, set with -ThrottleLimit), which helps when each round mostly waits, for example on a remote server. Each parallel block runs in its own runspace and cannot see your variables: you must pass them with the $using: prefix. Results arrive in whatever order they finish, so sort them if order matters.

$rate = 0.28
'with $using:'
4521, 312, 9870 | ForEach-Object -Parallel { '{0} kWh costs {1}' -f $_, [math]::Round($_ * $using:rate, 2) } |
    Sort-Object
'without $using:'
4521, 312, 9870 | ForEach-Object -Parallel { '{0} kWh costs {1}' -f $_, [math]::Round($_ * $rate, 2) } |
    Sort-Object

with $using:
312 kWh costs 87.36
4521 kWh costs 1265.88
9870 kWh costs 2763.6
without $using:
312 kWh costs 0
4521 kWh costs 0
9870 kWh costs 0

Without $using:, $rate is simply empty inside the block, and every cost silently becomes 0: no error at all. Bash scripts parallelize with xargs -P or by starting background jobs; see Remote Execution for running the same command on many servers.

Reading a file line by line

Processing a CSV export or a list of servers one line at a time is one of the most common loops, and in Bash it has the most traps. The safe pattern is the while read loop: a while loop whose condition is read -r, with the file redirected into the loop after done. read takes one line per round and fails at the end of the file, which ends the loop.

cat > readings.csv <<'EOF'
M-1042,4521
M-1043,312
  M-1044,9870
EOF
printf 'M-1045,128' >> readings.csv    # last line without a newline

echo "--- plain while read:"
while IFS=, read -r meter kwh; do
  echo "[$meter] [$kwh]"
done < readings.csv

echo "--- also keeping a last line without newline:"
while IFS=, read -r meter kwh || [[ -n $meter ]]; do
  echo "[$meter] [$kwh]"
done < readings.csv

--- plain while read:
[M-1042] [4521]
[M-1043] [312]
[  M-1044] [9870]
--- also keeping a last line without newline:
[M-1042] [4521]
[M-1043] [312]
[  M-1044] [9870]
[M-1045] [128]

• IFS=, sets the field separator for this read only, so each line is split at the comma; the last variable gets the rest of the line. With IFS= (empty) and one variable you get the whole line unchanged.
• With IFS=,, spaces are ordinary characters, so the leading spaces before M-1044 are kept. The default IFS would have trimmed them.
• -r keeps backslashes as they are (ShellCheck SC2162 warns without it).
• A file whose last line has no newline, common in exports from Windows tools, loses that line: read fills the variables but reports failure. || [[ -n $meter ]] keeps it.

Simple splitting at commas only works for simple CSV; quoted fields with commas inside need a real CSV parser (see JSON, CSV and APIs).

The subshell trap

In Bash, every part of a pipeline runs in its own subshell, a copy of the shell. A while loop at the end of a pipe therefore changes the copy's variables, and the copy disappears when the pipeline ends:

printf '%s\n' M-1042 M-1043 M-1044 > meters.txt

count=0
cat meters.txt | while read -r meter; do count=$(( count + 1 )); done
echo "after the pipe:      count=$count"

count=0
while read -r meter; do count=$(( count + 1 )); done < meters.txt
echo "after < redirection: count=$count"

mapfile -t meters < meters.txt
echo "mapfile read ${#meters[@]} lines: ${meters[*]}"

after the pipe:      count=0
after < redirection: count=3
mapfile read 3 lines: M-1042 M-1043 M-1044

When the input has to come from a command rather than a file, use process substitution, done < <(command), which keeps the loop in the main shell. If you just need all lines in an array, mapfile -t (also called readarray) reads them in one step.

The same in PowerShell

Get-Content returns the lines of a file as strings, a last line without a newline included. The ForEach-Object script block runs in the current scope, so a counter changed inside it keeps its value:

Set-Content -Path readings.csv -Value 'M-1042,4521', 'M-1043,312', 'M-1044,9870'

foreach ($line in Get-Content -Path readings.csv) {
    $meter, $kwh = $line -split ','
    if ([int]$kwh -gt 1000) { "$meter used $kwh kWh" }
}

$count = 0
Get-Content -Path readings.csv | ForEach-Object { $count++ }
"count after the pipeline: $count"

M-1042 used 4521 kWh
M-1044 used 9870 kWh
count after the pipeline: 3

$meter, $kwh = $line -split ',' assigns the parts of the split to two variables, much like IFS=, read. The [int] cast is needed because the parts are strings (see the left operand decides the type). For real CSV files, Import-Csv turns each row into an object with named properties.

Waiting and retrying: while and until

A while loop repeats as long as its condition is true; until (Bash) and do … until (PowerShell) repeat as long as it is false. Their typical job in a script is waiting: for a file to appear, for a service to come back after a restart, for a lock to be released. Such a loop needs two things: a pause between attempts, so it does not run thousands of checks a second, and a maximum, so a problem leads to a clear error instead of an infinite loop.

attempt=0
max_attempts=5
until [[ -f export.done ]]; do
  attempt=$(( attempt + 1 ))
  if (( attempt > max_attempts )); then
    echo "export.done did not appear after $max_attempts checks" >&2
    exit 1
  fi
  echo "check $attempt: waiting for export.done"
  (( attempt == 3 )) && touch export.done   # here the export finishes
  # a real script pauses here: sleep 10
done
echo "export finished, continuing"

check 1: waiting for export.done
check 2: waiting for export.done
check 3: waiting for export.done
export finished, continuing

$attempt = 0
$maxAttempts = 5
do {
    $attempt++
    if ($attempt -gt $maxAttempts) {
        throw "export.done did not appear after $maxAttempts checks"
    }
    "check ${attempt}: waiting for export.done"
    if ($attempt -eq 3) { New-Item -Path export.done -ItemType File | Out-Null }
    # a real script pauses here: Start-Sleep -Seconds 10
} until (Test-Path -Path export.done)
'export finished, continuing'

check 1: waiting for export.done
check 2: waiting for export.done
check 3: waiting for export.done
export finished, continuing

do { … } until (…) and do { … } while (…) test their condition after the block, so the body always runs at least once; while (…) { … } tests first. The ${attempt} braces stop PowerShell from reading the colon after the name as part of the variable. Both scripts fail with a clear message and a non-zero exit code when the maximum is reached, so a scheduler notices (see Exit codes).

break, continue and loop labels

continue skips the rest of the current round and break leaves the loop, in both shells (the general idea: break and continue). A patch round that skips a server in maintenance and stops when the change window ends:

for server in web01 web02 db01 db02; do
  if [[ $server == web02 ]]; then
    echo "skipping $server (in maintenance)"
    continue
  fi
  echo "patching $server"
  if [[ $server == db01 ]]; then
    echo "change window over, stopping"
    break
  fi
done

patching web01
skipping web02 (in maintenance)
patching db01
change window over, stopping

In nested loops, both commands act on the innermost loop. Bash accepts a number: break 2 leaves two levels. PowerShell uses a loop label instead: a name with a colon in front of a loop (:racks foreach …), which break racks or continue racks can target from any depth. Searching the first free slot in a rack:

for rack in A B; do
  for slot in 1 2 3; do
    if [[ $rack$slot == B2 ]]; then
      echo "free slot: $rack$slot"
      break 2
    fi
    echo "slot $rack$slot taken"
  done
done

slot A1 taken
slot A2 taken
slot A3 taken
slot B1 taken
free slot: B2

:racks foreach ($rack in 'A', 'B') {
    foreach ($slot in 1..3) {
        if ("$rack$slot" -eq 'B2') {
            "free slot: $rack$slot"
            break racks
        }
        "slot $rack$slot taken"
    }
}

slot A1 taken
slot A2 taken
slot A3 taken
slot B1 taken
free slot: B2

return and break inside ForEach-Object

ForEach-Object is not a loop but a cmdlet that calls a script block once per object, and that changes what the keywords do. return ends the current call of the script block, which behaves like continue:

1..5 | ForEach-Object {
    if ($_ -eq 3) { return }    # skips 3, like continue
    "item $_"
}
'after the pipeline'

item 1
item 2
item 4
item 5
after the pipeline

break and continue look for an enclosing loop instead. If there is none, they end the whole script, without an error message:

'before the pipeline'
1..5 | ForEach-Object {
    if ($_ -eq 3) { break }
    "item $_"
}
'after the pipeline: never printed'

before the pipeline
item 1
item 2

Side by side