Scheduling and Logging

What changes when nobody is watching

A script that works perfectly when Joost Bakker runs it from his terminal can fail at 02:30 when a scheduler starts it. Four things are different in an unattended run:

• The environment is small. cron starts jobs with only a few variables (such as HOME, LOGNAME and SHELL=/bin/sh) and usually a short PATH such as /usr/bin:/bin; nothing from your ~/.bashrc or PowerShell profile is loaded. Commands installed in ~/.local/bin or variables you export in your profile are simply not there. See the cron environment.
• The working directory is different. cron starts in the user's home directory, Task Scheduler often in C:\Windows\System32. A relative path such as ./readings.csv then points somewhere else.
• There is no terminal. A prompt (read, Read-Host, a confirmation question) waits forever or fails. Output nobody captures is lost.
• It may run as another account, with other permissions, other network drives and another home directory.

You can rehearse the first two problems without a scheduler. env -i starts a command with an empty environment, so you can hand the script roughly what cron would give it:

mkdir -p joost/kestrel
cat > joost/kestrel/report.sh <<'END'
#!/bin/bash
echo "KESTREL_ENV: ${KESTREL_ENV:-<not set>}"
if command -v shellcheck > /dev/null; then echo "shellcheck: found"; else echo "shellcheck: not found"; fi
echo "working directory: $(basename "$PWD")"
END
chmod +x joost/kestrel/report.sh
export KESTREL_ENV=production

echo "--- run by hand, from the script's folder"
(cd joost/kestrel && ./report.sh)

echo "--- run with an environment like cron's"
(cd joost && env -i PATH=/usr/bin:/bin SHELL=/bin/sh kestrel/report.sh)

--- run by hand, from the script's folder
KESTREL_ENV: production
shellcheck: found
working directory: kestrel
--- run with an environment like cron's
KESTREL_ENV: <not set>
shellcheck: not found
working directory: joost

Same script, three differences: the exported variable is gone, shellcheck (installed in ~/.local/bin) is no longer found, and the script starts in the home folder instead of its own. This is the classic "works when I run it, fails in cron" report.

Making a script schedule-ready

A few habits remove those surprises before the first scheduled run.

Start in the script's own folder

Let the script find its own location and move there, so relative paths to its data files work no matter who starts it. In Bash, $0 holds the path the script was started with; in PowerShell, the automatic variable $PSScriptRoot holds the folder of the running script.

mkdir -p joost/kestrel
cat > joost/kestrel/report.sh <<'END'
#!/bin/bash
cd "$(dirname "$0")" || exit 1
echo "working directory: $(basename "$PWD")"
printf 'M-1042;4521\n' > last-export.csv
END
chmod +x joost/kestrel/report.sh
cd joost && env -i PATH=/usr/bin:/bin kestrel/report.sh
ls kestrel

working directory: kestrel
last-export.csv
report.sh

Set-Location -Path $PSScriptRoot
"Script folder: $PSScriptRoot"
"Working dir:   $((Get-Location).Path)"

Script folder: /home/amara/scripts
Working dir:   /home/amara/scripts

The || exit 1 matters: if the cd fails, the script must stop rather than write files in the wrong place (ShellCheck warns about a bare cd, see Testing, Linting and Style).

The checklist

The exit code is how every scheduler knows whether a run went well, so it must be honest; see Conditions, Tests and Exit Codes and Error Handling. Note that pwsh -File exits with 0 after an ordinary (non-terminating) error: only exit N, an uncaught throw or an error with -ErrorAction Stop turn into a non-zero code. Locking against overlapping runs (a 15-minute job that sometimes takes 20) is on Writing Robust and Safe Scripts.

Scheduling on Linux: cron and systemd timers

How to write a crontab line and a timer unit is explained on Scheduling Tasks (the five fields, systemd timers, which to choose). What matters for the script is the command line you schedule. A robust crontab entry for Kestrel Energy's nightly meter export:

# m  h  dom mon dow  command
30   2  *   *   *    /usr/bin/flock -n /run/lock/kestrel-export.lock /opt/kestrel/bin/export.sh >> /var/log/kestrel/export.log 2>&1

• Absolute paths everywhere, because of the short cron PATH.
• flock -n skips the run if the previous one still holds the lock.
• >> … 2>&1 appends standard output and standard error to a log; otherwise cron tries to mail the output, and on most servers it is simply lost.
• A % in a crontab command means "new line" to cron; write \%, or better, keep date formats inside the script.

With a systemd timer you need no redirection: everything the script prints goes to the journal, and the exit code becomes the state of the service unit.

[Unit]
Description=Kestrel nightly meter export

[Service]
Type=oneshot
User=kestrel-export
ExecStart=/opt/kestrel/bin/export.sh

$ systemctl status kestrel-export.service --no-pager
$ journalctl -u kestrel-export.service --since today

× kestrel-export.service - Kestrel nightly meter export
     Loaded: loaded (/etc/systemd/system/kestrel-export.service; static)
     Active: failed (Result: exit-code) since Mon 2026-10-05 02:30:04 CEST; 6h ago
    Process: 48211 ExecStart=/opt/kestrel/bin/export.sh (code=exited, status=1/FAILURE)

Oct 05 02:30:01 bill-app-01 export.sh[48211]: 2026-10-05T00:30:01Z [INFO] billing export started
Oct 05 02:30:04 bill-app-01 export.sh[48211]: 2026-10-05T00:30:04Z [ERROR] database db01 not reachable
Oct 05 02:30:04 bill-app-01 systemd[1]: kestrel-export.service: Failed with result 'exit-code'.

The server's clock runs on Amsterdam time, so the export starts at 02:30 local time; the script stamps its own lines in UTC (00:30Z), as Log hygiene recommends.

Scheduling on Windows: Task Scheduler

Task Scheduler is the Windows service that starts programs at set times or on events (at logon, at startup, when an event is logged). Each job it runs is a scheduled task: a named definition with one or more triggers (when), one or more actions (what to run), the account to run as, and settings such as "stop the task if it runs longer than 1 hour". You can build one in the Task Scheduler window (taskschd.msc), with the older schtasks.exe, or in PowerShell with the ScheduledTasks module, which exists only on Windows:

# Windows only: register the nightly export as a scheduled task
$action = New-ScheduledTaskAction -Execute 'C:\Program Files\PowerShell\7\pwsh.exe' `
    -Argument '-NoProfile -NonInteractive -File C:\Scripts\Export-Readings.ps1' `
    -WorkingDirectory 'C:\Scripts'
$trigger = New-ScheduledTaskTrigger -Daily -At '02:30'
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Hours 1) `
    -MultipleInstances IgnoreNew

Register-ScheduledTask -TaskName 'Kestrel meter export' -TaskPath '\Kestrel\' `
    -Action $action -Trigger $trigger -Settings $settings `
    -User 'KESTREL\svc-export' -Password $env:SVC_EXPORT_PASSWORD

• -NoProfile -NonInteractive -File is the scheduled equivalent of the Bash checklist: no profile, no prompts, a full path.
• -MultipleInstances IgnoreNew (the default) is Task Scheduler's own protection against overlapping runs.
• Run the task under a dedicated service account with only the rights the job needs (in a domain, a group managed service account avoids storing a password at all). The password here comes from an environment variable for illustration; never type it into the script.
• Do not add -ExecutionPolicy Bypass by reflex; sign the script or set RemoteSigned instead (see Running Scripts).

Task Scheduler shows the script's exit code as Last Run Result: 0x0 is success, 0x1 means the script ended with exit code 1. You can read it from PowerShell too:

Get-ScheduledTask -TaskPath '\Kestrel\' | Get-ScheduledTaskInfo |
    Select-Object TaskName, LastRunTime, LastTaskResult, NextRunTime

TaskName             LastRunTime         LastTaskResult NextRunTime
--------             -----------         -------------- -----------
Kestrel meter export 10/6/2026 2:30:00 AM              1 10/7/2026 2:30:00 AM

Logging from a Bash script

A good log line answers when, how bad and what: a timestamp in UTC (in ISO 8601 form, so lines from different servers sort and compare), a level such as INFO, WARN or ERROR, and a message with the details someone needs at 08:00 (which meter, which server, which ticket). A small function keeps every line in the same shape:

export NOW=2026-10-06T09:00:00Z   # fixed clock for this page; leave it out in real use

log() {
  local level=$1; shift
  printf '%s [%s] %s\n' "$(date -u -d "${NOW:-now}" +%Y-%m-%dT%H:%M:%SZ)" "$level" "$*" |
    tee -a export.log >&2
}

log INFO "billing export started"
log WARN "meter M-1043 has no reading for 2026-10-05"
log ERROR "export failed: 1 of 1824 meters missing"
echo "--- export.log:"
cat export.log

2026-10-06T09:00:00Z [INFO] billing export started
2026-10-06T09:00:00Z [WARN] meter M-1043 has no reading for 2026-10-05
2026-10-06T09:00:00Z [ERROR] export failed: 1 of 1824 meters missing
--- export.log:
2026-10-06T09:00:00Z [INFO] billing export started
2026-10-06T09:00:00Z [WARN] meter M-1043 has no reading for 2026-10-05
2026-10-06T09:00:00Z [ERROR] export failed: 1 of 1824 meters missing

tee -a appends each line to the file and still shows it, and >&2 sends the copy to standard error. Log messages on standard error keep standard output free for the data the script produces, so ./export.sh > readings.csv gives a clean file. The NOW variable is only there so this page prints the same time on every run; it also happens to make the function testable.

To capture everything a script prints, including the error messages of the commands it calls, redirect the whole script once at the top with exec:

cat > nightly.sh <<'END'
#!/bin/bash
exec >> nightly.log 2>&1      # everything below goes to the log
echo "nightly export started"
ls missing-folder
echo "nightly export finished"
END
bash nightly.sh
echo "exit status: $?  (the terminal stayed silent)"
cat nightly.log

exit status: 0  (the terminal stayed silent)
nightly export started
ls: cannot access 'missing-folder': No such file or directory
nightly export finished

The log caught the ls error, but look at the exit status: 0, because the last command succeeded. A scheduler would report this run as fine. Logging does not replace error handling; see Error Handling.

Sending lines to the system log

logger is a standard Linux command that writes a message to the system log: on a systemd machine into the journal, elsewhere to the syslog daemon, which can forward it to a central log server. The -t option sets a tag to search for, -p a priority. It is handy for a single "started/finished/failed" line that monitoring should see. (Under a systemd timer, plain output already ends up in the journal; systemd-cat does the same for a command started elsewhere.)

$ logger -t kestrel-export -p user.err "export failed: database db01 not reachable"
$ journalctl -t kestrel-export --since today

Oct 05 02:30:04 bill-app-01 kestrel-export[48230]: export failed: database db01 not reachable

Logging from a PowerShell script

The same function in PowerShell. It appends to the file with Add-Content and shows the line with Write-Host, which writes to the Information stream instead of the output. That matters: anything a function writes to the success stream becomes part of its return value, so a log function that used Write-Output (or Tee-Object) would mix log lines into the data of every function that calls it.

$env:NOW = '2026-10-06T09:00:00Z'   # fixed clock for this page; leave it out in real use

function Write-Log {
    param(
        [ValidateSet('INFO', 'WARN', 'ERROR')] [string] $Level,
        [string] $Message
    )
    $stamp = if ($env:NOW) { $env:NOW } else {
        [datetime]::UtcNow.ToString("yyyy-MM-dd'T'HH:mm:ss'Z'")
    }
    $line = "$stamp [$Level] $Message"
    Add-Content -Path ./export.log -Value $line
    Write-Host $line
}

Write-Log -Level INFO -Message 'billing export started'
Write-Log -Level WARN -Message 'meter M-1043 has no reading for 2026-10-05'
'--- export.log:'
Get-Content -Path ./export.log

2026-10-06T09:00:00Z [INFO] billing export started
2026-10-06T09:00:00Z [WARN] meter M-1043 has no reading for 2026-10-05
--- export.log:
2026-10-06T09:00:00Z [INFO] billing export started
2026-10-06T09:00:00Z [WARN] meter M-1043 has no reading for 2026-10-05

Transcripts

A transcript is PowerShell's built-in session recorder: between Start-Transcript and Stop-Transcript, everything that appears in the console (output, warnings, errors, Write-Host text) is also written to a text file, with a header naming the computer, the user and the start time. It is the quickest way to keep a record of a scheduled run, though less tidy than your own log lines. Add -Append to keep earlier runs, or put a date in the file name.

Start-Transcript -Path ./export-run.log
Write-Host 'Exporting 1824 meter readings'
Write-Warning 'meter M-1043 has no reading'
Stop-Transcript
'Lines mentioning M-1043 in the transcript:'
(Select-String -Path ./export-run.log -Pattern 'M-1043').Line

Transcript started, output file is ./export-run.log
Exporting 1824 meter readings
WARNING: meter M-1043 has no reading
Transcript stopped, output file is /home/amara/scripts/export-run.log
Lines mentioning M-1043 in the transcript:
WARNING: meter M-1043 has no reading

On Windows, scripts can also write to the Windows event log, where monitoring tools already look. The Write-EventLog cmdlet exists only in Windows PowerShell 5.1; PowerShell 7 removed the *-EventLog cmdlets and offers New-WinEvent (and Get-WinEvent to read) on Windows. On Linux, PowerShell itself logs to syslog.

Structured logging: one JSON object per line

Text lines are easy to read but awkward to filter: "all errors for meter M-1077 this week" needs a regular expression that breaks when someone changes the wording. In structured logging each event is written as a record with named fields (timestamp, level, message, and context such as the meter or server), usually as one JSON object per line. Log platforms (the ELK stack, Splunk, Azure Monitor, Grafana Loki) read such lines without any parsing rules.

log_json() {
  jq -nc --arg ts "$(date -u -d "${NOW:-now}" +%Y-%m-%dT%H:%M:%SZ)" \
    --arg level "$1" --arg msg "$2" --arg meter "$3" \
    '{ts: $ts, level: $level, msg: $msg, meter: $meter}'
}
export NOW=2026-10-06T09:00:00Z   # fixed clock for this page
log_json WARN "no reading" M-1043 >> export.jsonl
log_json ERROR "reading is negative" M-1077 >> export.jsonl
cat export.jsonl
echo "--- only errors:"
jq -r 'select(.level == "ERROR") | "\(.meter): \(.msg)"' export.jsonl

{"ts":"2026-10-06T09:00:00Z","level":"WARN","msg":"no reading","meter":"M-1043"}
{"ts":"2026-10-06T09:00:00Z","level":"ERROR","msg":"reading is negative","meter":"M-1077"}
--- only errors:
M-1077: reading is negative

Letting jq --arg build the object, instead of gluing a JSON string together with printf, guarantees valid JSON even when a message contains quotes or backslashes. PowerShell builds the same lines from an ordered hash table:

function Write-JsonLog {
    param([string] $Level, [string] $Message, [string] $Meter)
    [ordered]@{
        ts    = '2026-10-06T09:00:00Z'   # real use: [datetime]::UtcNow.ToString('o')
        level = $Level
        msg   = $Message
        meter = $Meter
    } | ConvertTo-Json -Compress | Add-Content -Path ./export.jsonl
}
Write-JsonLog -Level WARN -Message 'no reading' -Meter M-1043
Write-JsonLog -Level ERROR -Message 'reading is negative' -Meter M-1077
Get-Content -Path ./export.jsonl
'--- only errors:'
Get-Content -Path ./export.jsonl | ConvertFrom-Json |
    Where-Object level -eq 'ERROR' |
    ForEach-Object { "$($_.meter): $($_.msg)" }

{"ts":"2026-10-06T09:00:00Z","level":"WARN","msg":"no reading","meter":"M-1043"}
{"ts":"2026-10-06T09:00:00Z","level":"ERROR","msg":"reading is negative","meter":"M-1077"}
--- only errors:
M-1077: reading is negative

More on jq, ConvertTo-Json and the JSON Lines format in JSON, CSV and APIs.

Log hygiene

• Never log secrets. Passwords, API tokens and connection strings do not belong in a log line, a transcript or set -x output. A transcript records everything on screen, so do not print a secret there either. Log that you authenticated, not with what.
• Be careful with personal data. A customer name or address in a log is personal data under the GDPR; log the meter or customer number instead.
• Use UTC. Local time jumps twice a year and differs between servers; UTC timestamps line up when you correlate logs.
• Rotate. A log that only grows fills the disk one day. Let logrotate handle files in /var/log (see log rotation), put the date in the file name, or delete logs older than a set number of days in the script itself.
• Log the start, the end and every decision. "Started", "skipped meter M-1043: no reading", "finished: 1823 exported, 1 skipped" lets the next person reconstruct a run without reading the code.