Writing Robust and Safe Scripts

Why scripts need to be robust

A script written for one afternoon tends to live for years. It gets scheduled, copied to other servers, run by colleagues who did not write it, interrupted halfway by a reboot, and started twice by accident. Each of those situations is harmless for a robust script and an incident for a fragile one. The habits on this page all answer one question: what happens if this runs at the wrong moment, with the wrong input, or for the second time?

They build on Error Handling (stop on failure, report it) and on the general advice on working safely on production systems.

Idempotency: safe to run twice

An idempotent script leaves the system in the same state whether it runs once or ten times: the second run finds the work done and changes nothing (the general idea is idempotence). That is what makes it safe to rerun after a failure halfway, or when nobody is sure whether last night's run happened. The trick is to describe the desired state ("this folder exists", "this line is in the file") instead of an action ("create the folder", "append the line"):

setup() {
  mkdir -p backups
  grep -qxF 'server=192.0.2.10' app.conf 2>/dev/null || echo 'server=192.0.2.10' >> app.conf
}
setup
setup            # second run: nothing changes
cat app.conf
echo "--- plain mkdir, second time:"
mkdir backups

server=192.0.2.10
--- plain mkdir, second time:
mkdir: cannot create directory ‘backups’: File exists

mkdir -p succeeds when the folder already exists; grep -qxF checks for the exact line (-x whole line, -F no regular expression, -q quiet) and only if it is missing does || append it. A plain mkdir, by contrast, fails on the second run. PowerShell has the same tools: -Force on New-Item and a Test-Path or Select-String check before adding:

function Initialize-ExportFolder {
    New-Item -Path ./backups -ItemType Directory -Force | Out-Null
    $line = 'server=192.0.2.10'
    if (-not (Test-Path -Path ./app.conf) -or
        -not (Select-String -Path ./app.conf -Pattern $line -SimpleMatch -Quiet)) {
        Add-Content -Path ./app.conf -Value $line
    }
}
Initialize-ExportFolder
Initialize-ExportFolder
Get-Content -Path ./app.conf
(Get-ChildItem -Directory).Name

server=192.0.2.10
backups

Some actions cannot be made idempotent by themselves, such as "send the invoice email" or "post the reading to the billing API". For those, record what was done (a marker file, a status column, a request ID the API recognizes) and check the record first.

Dry runs

Before a script deletes, moves or changes something for real, you want to see what it would do: a dry run. Bash has no built-in mechanism, but a small wrapper function gives you one. Every risky command goes through run, which either prints it or executes it. The safe mode is the default; the real run needs an explicit DRY_RUN=0:

cat > purge-exports.sh <<'END'
#!/bin/bash
DRY_RUN=${DRY_RUN:-1}          # safe default: only show what would happen
run() {
  if [[ $DRY_RUN == 1 ]]; then
    echo "would run: $*"
  else
    "$@"
  fi
}
for f in export-2026-09-*.csv; do
  run rm -- "$f"
done
END
touch export-2026-09-01.csv export-2026-09-02.csv export-2026-10-01.csv
bash purge-exports.sh
echo "--- for real:"
DRY_RUN=0 bash purge-exports.sh
ls

would run: rm -- export-2026-09-01.csv
would run: rm -- export-2026-09-02.csv
--- for real:
export-2026-10-01.csv
purge-exports.sh

"$@" runs the arguments exactly as given, each one a separate word, so file names with spaces survive. The -- after rm ends its options, so a file called -rf cannot be mistaken for one.

-WhatIf and ShouldProcess in PowerShell

PowerShell builds dry runs into the language. Cmdlets that change things accept the common parameter -WhatIf, which makes them report what they would do, on which target, and then do nothing:

Set-Content -Path ./old.log -Value 'x'
Remove-Item -Path ./old.log -WhatIf
"Still there: $(Test-Path -Path ./old.log)"

What if: Performing the operation "Remove File" on target "/home/amara/scripts/old.log".
Still there: True

Your own functions get the same behavior through ShouldProcess. Declare [CmdletBinding(SupportsShouldProcess)], and the function automatically accepts -WhatIf and -Confirm. Then wrap every change in if ($PSCmdlet.ShouldProcess(target, action)): the call returns $true in a normal run, prints a "What if" line and returns $false under -WhatIf, and asks the user under -Confirm. It also passes -WhatIf on to the cmdlets you call inside.

function Remove-OldLog {
    [CmdletBinding(SupportsShouldProcess)]
    param([Parameter(Mandatory)] [string[]] $Path)
    foreach ($file in $Path) {
        if ($PSCmdlet.ShouldProcess($file, 'Delete log file')) {
            Remove-Item -LiteralPath $file
            "Deleted $file"
        }
    }
}
'app-2026-09-01.log', 'app-2026-09-02.log' | ForEach-Object { Set-Content -Path $_ -Value 'x' }

Remove-OldLog -Path 'app-2026-09-01.log', 'app-2026-09-02.log' -WhatIf
"Files left: $((Get-ChildItem -Filter *.log | Sort-Object Name).Name -join ', ')"
Remove-OldLog -Path 'app-2026-09-01.log'
"Files left: $((Get-ChildItem -Filter *.log).Name -join ', ')"

What if: Performing the operation "Delete log file" on target "app-2026-09-01.log".
What if: Performing the operation "Delete log file" on target "app-2026-09-02.log".
Files left: app-2026-09-01.log, app-2026-09-02.log
Deleted app-2026-09-01.log
Files left: app-2026-09-02.log

For really destructive functions, add ConfirmImpact = 'High' to CmdletBinding. Because the default $ConfirmPreference is High, PowerShell then asks for confirmation even without -Confirm; a scheduled job passes -Confirm:$false deliberately. More about advanced functions on Functions and Parameters.

Validating input

Every value that comes from outside the script (an argument, a CSV column, an API response, a file name) can be empty, malformed or hostile. Check it against what you expect, as narrowly as possible, before you use it; this is input validation and it applies to admin scripts as much as to web forms. A meter ID at Kestrel Energy is M- and four digits, so accept exactly that and nothing else:

cat > meter-report.sh <<'END'
#!/bin/bash
id=${1-}
if [[ ! $id =~ ^M-[0-9]{4}$ ]]; then
  echo "usage: meter-report.sh M-nnnn (got: '$id')" >&2
  exit 2
fi
echo "report for $id"
END
bash meter-report.sh M-1042
bash meter-report.sh '../../etc/passwd'
echo "exit status: $?"

report for M-1042
usage: meter-report.sh M-nnnn (got: '../../etc/passwd')
exit status: 2

In PowerShell, the same rule is a validation attribute on the parameter, and PowerShell rejects the value before the function body runs:

function Get-MeterReport {
    param([Parameter(Mandatory)] [ValidatePattern('^M-\d{4}$')] [string] $MeterId)
    "report for $MeterId"
}
Get-MeterReport -MeterId M-1042
Get-MeterReport -MeterId '../../etc/passwd'

report for M-1042
Get-MeterReport: /home/amara/scripts/example.ps1:6
Line |
   6 |  Get-MeterReport -MeterId '../../etc/passwd'
     |                           ~~~~~~~~~~~~~~~~~~
     | Cannot validate argument on parameter 'MeterId'. The argument
     | "../../etc/passwd" does not match the "^M-\d{4}$" pattern. Supply an
     | argument that matches "^M-\d{4}$" and try the command again.

Prefer a whitelist (what is allowed) over a blacklist (what is forbidden): you cannot think of every bad value, but you know exactly what a good one looks like. ValidateSet, ValidateRange and ValidateScript cover most other cases.

Guarding deletions

The most dangerous line in shell scripting is a delete built from a variable. If $dir is empty (a typo, a failed lookup, a missing argument), rm -rf "$dir/"* becomes rm -rf /*. The expansion ${dir:?} stops the script with an error when the variable is unset or empty, so the rm never runs (ShellCheck warns about the unguarded form as SC2115):

cat > cleanup.sh <<'END'
#!/bin/bash
dir=""                       # imagine a typo or a failed lookup
rm -rf "${dir:?}/"*
echo "never printed"
END
bash cleanup.sh
echo "exit status: $?"

cleanup.sh: line 3: dir: parameter null or not set
exit status: 1

In PowerShell, use -LiteralPath for paths that come from data (so [ and * are not treated as wildcards), and check that a path is inside the folder you expect before deleting anything under it.

Temporary files

Scripts often need a scratch file: a sorted copy, a download, an intermediate result. Two things go wrong with /tmp/export.tmp: two runs (or two users) write the same file, and the file stays behind when the script fails. On a shared server a predictable name in /tmp is also a security risk, because another user can create it first. mktemp creates a new file (or, with -d, a directory) with a random, unused name, readable only by you, and prints that name. Pair it with a trap on EXIT, which runs when the script ends for any reason, including an error under set -e:

cat > convert.sh <<'END'
#!/bin/bash
set -euo pipefail
tmp=$(mktemp)
trap 'rm -f "$tmp"; echo "cleaned up temp file"' EXIT
echo "$tmp" > last-tmp-name.txt
printf 'M-1043;312\nM-1042;4521\n' | sort > "$tmp"
cp "$tmp" sorted.csv
ls missing-input.csv            # fails: set -e ends the script here
echo "never printed"
END
bash convert.sh
echo "exit status: $?"
[[ -e $(cat last-tmp-name.txt) ]] && echo "temp file still there" || echo "temp file is gone"
cat sorted.csv

ls: cannot access 'missing-input.csv': No such file or directory
cleaned up temp file
exit status: 2
temp file is gone
M-1042;4521
M-1043;312

The script failed, kept its exit status (2, from ls), and still removed its temporary file. In PowerShell, New-TemporaryFile creates the file and a finally block does the cleanup, because finally runs whether the try block succeeded or failed:

$tmp = New-TemporaryFile
try {
    'M-1043;312', 'M-1042;4521' | Sort-Object | Set-Content -Path $tmp.FullName
    Copy-Item -Path $tmp.FullName -Destination ./sorted.csv
    Get-Content -Path ./missing-input.csv -ErrorAction Stop
}
catch {
    "Failed: $($_.Exception.Message)"
}
finally {
    Remove-Item -LiteralPath $tmp.FullName
    "Temp file still there: $(Test-Path -LiteralPath $tmp.FullName)"
}
Get-Content -Path ./sorted.csv

Failed: Cannot find path '/home/amara/scripts/missing-input.csv' because it does not exist.
Temp file still there: False
M-1042;4521
M-1043;312

Note the -ErrorAction Stop: without it, Get-Content's error is non-terminating and the catch block would never run (see Error Handling).

Locking: one run at a time

A job scheduled every 15 minutes that once takes 20 starts a second copy while the first is still busy. Two exports writing the same file, or two cleanups deleting the same folder, is a race condition. The fix is a lock: the first run takes it, and a second run that cannot get it stops.

On Linux, flock (from util-linux) takes an advisory lock on a file. The lock belongs to an open file descriptor, so it is released automatically when the script exits, even after a crash or kill -9: no stale lock files to clean up. With -n it does not wait but fails at once when the lock is taken. The example plays the long-running first instance by holding the lock in the outer shell:

cat > export.sh <<'END'
#!/bin/bash
exec 9> kestrel-export.lock
if ! flock -n 9; then
  echo "$1: another export is running, exiting" >&2
  exit 1
fi
echo "$1: lock acquired, exporting"
END
exec 8> kestrel-export.lock
flock -n 8               # simulate a long-running first instance holding the lock
bash export.sh "second instance"
echo "exit status: $?"
exec 8>&-                # first instance ends: lock released
bash export.sh "third instance"

second instance: another export is running, exiting
exit status: 1
third instance: lock acquired, exporting

exec 9> file opens the lock file on file descriptor 9 for the rest of the script, and flock -n 9 locks it. For a crontab line you do not even have to change the script: flock -n /run/lock/export.lock /opt/kestrel/bin/export.sh wraps the whole command (see Scheduling and Logging). Whether a second run should exit with 0 ("nothing to do") or 1 ("could not run") depends on whether you want an alert.

PowerShell has no lock cmdlet. A portable way is to open a lock file with sharing set to None: a second open, in this or another process, fails until the first handle is closed. On Linux .NET implements this with the same kind of advisory lock, and it is released when the process ends.

function Open-Lock([string] $Path) {
    try {
        [System.IO.File]::Open($Path, 'OpenOrCreate', 'ReadWrite', 'None')
    }
    catch [System.IO.IOException] {
        $null
    }
}
$first = Open-Lock ./kestrel-export.lock
"First instance got the lock:  $($null -ne $first)"
$second = Open-Lock ./kestrel-export.lock
"Second instance got the lock: $($null -ne $second)"
$first.Dispose()
$third = Open-Lock ./kestrel-export.lock
"After release, got the lock:  $($null -ne $third)"
$third.Dispose()

First instance got the lock:  True
Second instance got the lock: False
After release, got the lock:  True

On Windows, a scheduled task can also rely on Task Scheduler's own "do not start a new instance" setting, and Windows code often uses a named mutex ([System.Threading.Mutex]) instead of a file.

Keeping secrets out of scripts

Passwords, API tokens and connection strings do not belong in a script. Scripts end up in Git, in tickets, in emails to colleagues and in backups, and a secret in any of those must be treated as leaked (see what never belongs in Git). Read secrets at run time instead, from one of these, roughly from simplest to best:

• An environment variable set by whatever starts the script (a systemd unit's EnvironmentFile=, a CI pipeline's secret store). Simple, but visible to processes of the same user.
• A file with 600 permissions, owned by the account the job runs as, outside the script's folder and outside Git.
• A secrets manager (Azure Key Vault, AWS Secrets Manager, HashiCorp Vault), queried with the job's own identity, so that access is logged and the secret can be rotated in one place.

#!/bin/bash
# Fail early and clearly when the secret is missing; never print it.
: "${KESTREL_API_TOKEN:?KESTREL_API_TOKEN is not set}"
# printf is a shell builtin, so the token never shows up in ps;
# -H @- makes curl (7.55 or later) read the header from standard input.
printf 'Authorization: Bearer %s\n' "$KESTREL_API_TOKEN" |
    curl -fsS -H @- https://api.kestrel.example/v1/readings

Beware of tracing. set -x prints every command after expansion, so the secret appears in the terminal, the log or the CI output:

export KESTREL_API_TOKEN='placeholder-not-a-real-token'
set -x
printf 'Authorization: Bearer %s\n' "$KESTREL_API_TOKEN" > /dev/null
set +x

+ printf 'Authorization: Bearer %s\n' placeholder-not-a-real-token
+ set +x

Switch tracing off (set +x) around lines that use secrets, and never put a secret on a command line: arguments are visible to other users in ps.

Secrets in PowerShell

SecretManagement (the Microsoft.PowerShell.SecretManagement module) gives scripts one set of commands, Get-Secret, Set-Secret and Register-SecretVault, in front of different storage back ends, called vaults: the local SecretStore module, Azure Key Vault, KeePass and others. The script asks for a secret by name and does not care where it is stored. As of October 2026 Microsoft calls SecretManagement and SecretStore feature complete: the repositories are archived and the last versions (1.1.2 and 1.0.6) date from 2022. They still work; for new work, consider calling your organization's secrets manager directly (for example the Az.KeyVault module).

# Once, interactively: install the modules and register a vault
Install-PSResource -Name Microsoft.PowerShell.SecretManagement, Microsoft.PowerShell.SecretStore
Register-SecretVault -Name KestrelVault -ModuleName Microsoft.PowerShell.SecretStore -DefaultVault
Set-Secret -Name BillingApiToken      # prompts for the value; nothing in the script

# In the script
$token = Get-Secret -Name BillingApiToken      # returns a SecureString
Invoke-RestMethod -Uri 'https://api.kestrel.example/v1/readings' `
    -Authentication Bearer -Token $token

Least privilege

A script can only damage what its account may touch. Apply least privilege:

• Do not run a whole script with sudo or as Administrator because one step needs it. Run the script as a normal account and elevate the one command that needs it, or grant that one command in sudoers.
• Give scheduled jobs their own service account, with access only to the folders, databases and servers that job uses.
• Make requirements explicit, so a script fails at the start instead of halfway. PowerShell's #Requires statements stop a script before it runs: #Requires -Version 7.4, #Requires -Modules Az.KeyVault, #Requires -RunAsAdministrator (Windows). In Bash, test at the top: [[ $EUID -eq 0 ]] && { echo "do not run as root" >&2; exit 1; }.
• Turn on the strict modes: set -euo pipefail in Bash (with its known exceptions) and Set-StrictMode -Version Latest plus $ErrorActionPreference = 'Stop' in PowerShell, so typos and failures stop the script instead of silently doing the wrong thing (see Error Handling).

Checklist

Tools can check several of these habits for you; see Testing, Linting and Style.