Remote Execution

Why run things remotely

Tomasz Nowak in the NOC has to check free disk space on twelve web servers before a release. Logging in to each one is slow and error-prone; a script that asks all twelve and prints one table is not. Remote execution means a script on one machine runs commands on others and collects the results. On Linux and network gear the tool is SSH (Secure Shell); on Windows it is PowerShell Remoting, which since PowerShell 6 can also use SSH.

This page is about using them from scripts. Setting up keys, the agent, ~/.ssh/config and jump hosts is explained on SSH and Remote Work, and the remote access methods themselves on Remote Access.

One command over ssh

Give ssh a command after the host name and it runs that command instead of an interactive shell, prints its output locally and exits with the command's exit status (or 255 if ssh itself failed to connect). That makes it usable in if statements:

$ ssh tomasz@192.0.2.21 'df -h /var'
$ if ssh web01 'test -f /etc/kestrel/maintenance'; then echo "web01 is in maintenance"; fi

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        40G   31G  7.2G  82% /var

Quoting for the remote shell

ssh does not pass your arguments to the remote command one by one. It joins them into a single string with spaces and hands that string to the remote user's shell, which splits it into words again. A file name with a space survives your local quoting and then breaks on the other side. printf %q quotes a value so that a shell reads it back as exactly one word. The example lets a second local bash -c play the remote shell:

file="meter readings.csv"
# What the remote shell would receive, unquoted and quoted:
printf 'naive:  ssh web01 %s\n' "wc -l $file"
printf 'quoted: ssh web01 %s\n' "wc -l $(printf '%q' "$file")"
# Let a local shell play the remote shell to see the difference:
printf 'M-1042;4521\nM-1043;312\n' > "$file"
bash -c "wc -l $file"
bash -c "wc -l $(printf '%q' "$file")"

naive:  ssh web01 wc -l meter readings.csv
quoted: ssh web01 wc -l meter\ readings.csv
wc: meter: No such file or directory
wc: readings.csv: No such file or directory
0 total
2 meter readings.csv

The same rule makes remote commands dangerous with untrusted input: a value such as x; rm -rf ~ becomes a second command on the server. Validate input first (see Writing Robust and Safe Scripts) and quote with %q. Keep the remote side simple: one command, or a script that already exists on the server.

Running a local script on a remote machine

For more than a one-liner, do not build a long command string. Send the script itself: bash -s on the remote side reads the script from standard input, and arguments after -- become its $1, $2, …

$ ssh web01 'bash -s -- 90' < check-disk.sh

WARNING: /var at 91% (threshold 90%)

The script needs no copy on the server, and it always runs the version you have in front of you (and in Git).

Non-interactive ssh

A script cannot type a password or answer "Are you sure you want to continue connecting?". Four settings make ssh safe to use unattended:

• Key-based login. An SSH key pair for the job, its public key in the server's authorized_keys. A scheduled job uses a key without a passphrase stored with 600 permissions, or an SSH agent; an interactive script can use your agent.
• BatchMode. The client option -o BatchMode=yes tells ssh never to ask for anything (password, passphrase, host key confirmation). If key login does not work, ssh fails at once with exit status 255 instead of hanging until someone kills the job.
• A connection timeout. -o ConnectTimeout=5: an unreachable server costs five seconds, not the operating system's default of a minute or more.
• Known host keys. Load the servers' host keys into known_hosts in advance (from your inventory, or check the fingerprints once). StrictHostKeyChecking=accept-new accepts a key for a host it has never seen but still refuses a changed key; never use StrictHostKeyChecking=no, which switches off the protection against a server being impersonated.

$ ssh -o BatchMode=yes -o ConnectTimeout=5 -i ~/.ssh/kestrel_report svc-report@web01 'uptime'
$ echo "exit status: $?"

svc-report@web01: Permission denied (publickey).
exit status: 255

Put the options in ~/.ssh/config under a Host block (see client config) so every script gets them. On the server you can restrict what a job's key may do: in authorized_keys, the options restrict, from="198.51.100.5" and command="/usr/local/bin/report.sh" limit the key to one source address and one command.

Looping over servers

The obvious script reads a list of servers and runs ssh for each. It has a famous trap: ssh passes its standard input to the remote command, so it reads, and swallows, the rest of the server list that the while read loop was reading from. The loop then ends after the first server. The example shows the effect with a stand-in function that, like ssh, reads standard input:

printf 'web01\nweb02\ndb01\n' > servers.txt

check() {            # stands in for ssh: like ssh, it reads standard input
  cat > /dev/null
  echo "checked $1"
}

echo "--- loop 1"
while read -r host; do
  check "$host"
done < servers.txt

echo "--- loop 2 (stdin of check redirected)"
while read -r host; do
  check "$host" < /dev/null
done < servers.txt

--- loop 1
checked web01
--- loop 2 (stdin of check redirected)
checked web01
checked web02
checked db01

With real ssh, add -n (which redirects its standard input from /dev/null) or < /dev/null. Also collect each server's exit status, so one failing host does not disappear in the output:

#!/bin/bash
failed=0
while read -r host; do
  if ssh -n -o BatchMode=yes -o ConnectTimeout=5 "$host" 'df --output=pcent / | tail -n 1'; then
    :
  else
    echo "$host: FAILED (status $?)" >&2
    failed=$((failed + 1))
  fi
done < servers.txt
exit $(( failed > 0 ))

Fan-out: many servers at once

One server after another is fine for twelve machines, but slow for five hundred. Fan-out is running the same command on many machines in parallel and collecting the results. In Bash, xargs -P starts several ssh processes at once (see xargs); output then arrives in whatever order servers answer, so label each line with the host and sort afterward:

$ xargs -P 10 -I{} ssh -n -o BatchMode=yes {} 'echo "{} $(df --output=pcent / | tail -n 1)"' < servers.txt | sort

Past a few dozen servers, or when the task is "make sure this configuration is in place" rather than "run this command", use a configuration management tool such as Ansible. It does the fan-out, the inventory, the error reporting and the idempotence (see playbooks) that a loop of ssh calls has to do by hand.

PowerShell Remoting

PowerShell Remoting runs PowerShell commands on other computers and returns the results as objects. It uses the PowerShell Remoting Protocol on top of one of two transports: WinRM (Windows Remote Management, Microsoft's implementation of the WS-Management web services standard, listening on TCP port 5985 for HTTP and 5986 for HTTPS), or SSH. In an Active Directory domain WinRM authenticates with Kerberos and encrypts the session even on port 5985. Remoting over WinRM is enabled by default on Windows Server 2012 and later; on Windows clients an administrator turns it on with Enable-PSRemoting.

Invoke-Command

Invoke-Command sends a script block to one or more computers, runs it there, and returns the output. With several computer names it fans out by itself: by default it works on 32 computers at a time (-ThrottleLimit), and every returned object gets a PSComputerName property, so you know where each row came from.

$servers = 'web01', 'web02', 'web03', 'db01'
Invoke-Command -ComputerName $servers -ScriptBlock {
    Get-Volume -DriveLetter C |
        Select-Object @{ n = 'FreePct'; e = { [math]::Round(100 * $_.SizeRemaining / $_.Size) } }
} | Sort-Object PSComputerName | Format-Table PSComputerName, FreePct

PSComputerName FreePct
-------------- -------
db01                41
web01               18
web02               35
web03               33

A computer that cannot be reached produces a (non-terminating) error for that computer, and the others still answer. Use -ErrorVariable to collect the failures, or compare the PSComputerName values with your list.

Passing local values

The script block runs in a different process on a different machine, so it cannot see your local variables. There are two ways to send them along: -ArgumentList with a param() block, or the $using: scope modifier, which copies a local variable's value into the remote script block. $using: only works when the block really runs remotely (or in a job or ForEach-Object -Parallel); run locally, PowerShell refuses it:

$threshold = 90
Invoke-Command -ScriptBlock {
    param($Limit)
    "Checking disks against $Limit%"
} -ArgumentList $threshold

Invoke-Command -ScriptBlock { "Limit inside: $using:threshold" }

Checking disks against 90%
InvalidOperation: /home/amara/scripts/example.ps1:7
Line |
   7 |  Invoke-Command -ScriptBlock { "Limit inside: $using:threshold" }
     |                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | A Using variable cannot be retrieved. A Using variable can be used only
     | with Invoke-Command, Start-Job, or InlineScript in the script workflow.
     | When it is used with Invoke-Command, the Using variable is valid only if
     | the script block is invoked on a remote computer.

In a remote call, $using:threshold works and is the shortest form: Invoke-Command -ComputerName web01 -ScriptBlock { "Limit: $using:threshold" }. To run a whole local script file on remote computers, use Invoke-Command -ComputerName $servers -FilePath .\Check-Disk.ps1; like bash -s, it sends the script's contents.

Sessions

Every Invoke-Command -ComputerName call builds a connection, runs the block and tears the connection down. A PSSession is a persistent connection to a remote computer: a PowerShell process on that computer that stays alive between commands, keeping its variables and loaded modules. Create it once, reuse it, and close it:

$session = New-PSSession -ComputerName db01
Invoke-Command -Session $session -ScriptBlock { $svc = Get-Service -Name MSSQLSERVER }
Invoke-Command -Session $session -ScriptBlock { $svc.Status }    # $svc still exists
Copy-Item -Path .\backup-check.sql -Destination C:\Temp\ -ToSession $session
Remove-PSSession -Session $session

For interactive work, Enter-PSSession -ComputerName db01 gives you a remote prompt ([db01]: PS C:\>), the PowerShell counterpart of logging in with ssh; Exit-PSSession leaves it. Scripts use Invoke-Command, not Enter-PSSession.

Remoting over SSH

Since PowerShell 6, remoting can also run over SSH, which works between Windows, Linux and macOS and needs no WinRM. Use -HostName instead of -ComputerName, with -UserName and -KeyFilePath. Both sides need PowerShell 6 or later and an SSH server; on the server, sshd_config needs a line that starts PowerShell as an SSH subsystem (for example Subsystem powershell /usr/bin/pwsh -sshs -NoLogo on Linux). OpenSSH is part of Windows 10 1809 and Windows Server 2019 and later, and installed by default from Windows Server 2025.

Invoke-Command -HostName 192.0.2.21 -UserName tomasz -KeyFilePath ~/.ssh/id_ed25519 `
    -ScriptBlock { Get-Content -Path /etc/os-release -TotalCount 1 }

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

On Linux and macOS, PowerShell usually has no WinRM client at all, so -ComputerName fails before it tries the network. This is what PowerShell 7.6 on a Linux machine prints:

Invoke-Command -ComputerName 192.0.2.10 -ScriptBlock { Get-Uptime }

Invoke-Command: /home/amara/scripts/example.ps1:1
Line |
   1 |  Invoke-Command -ComputerName 192.0.2.10 -ScriptBlock { Get-Uptime }
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | This parameter set requires WSMan, and no supported WSMan client library
     | was found. WSMan is either not installed or unavailable for this system.

More on what differs between PowerShell on Windows and elsewhere: Cross-Platform Scripting.

Remote results are copies

Objects cannot travel over the network as live .NET objects. PowerShell serializes them to XML on the remote side and rebuilds them locally as deserialized objects: property bags with the same property values, a type name starting with Deserialized., and no methods. So $service.Stop() on a returned service object does nothing useful; call Stop-Service inside the script block instead. You can watch the same conversion locally:

Set-Content -Path ./readings.csv -Value 'M-1042;4521'
$local = Get-Item -Path ./readings.csv
$xml = [System.Management.Automation.PSSerializer]::Serialize($local)
$copy = [System.Management.Automation.PSSerializer]::Deserialize($xml)
"Local type:   $($local.GetType().FullName)"
"Copy type:    $($copy.pstypenames[0])"
"Name, Length: $($copy.Name), $($copy.Length)"
"Local has CopyTo(): $($null -ne $local.PSObject.Methods['CopyTo'])"
"Copy has CopyTo():  $($null -ne $copy.PSObject.Methods['CopyTo'])"

Local type:   System.IO.FileInfo
Copy type:    Deserialized.System.IO.FileInfo
Name, Length: readings.csv, 12
Local has CopyTo(): True
Copy has CopyTo():  False

The rule of thumb: do the work remotely, and return only the data you need (filter and select inside the script block). That is also faster, because less data crosses the network. ssh has the same rule in a text form: whatever the remote command prints is all you get back.

The second hop problem

Ravi Kumar connects from his workstation to web01 with Invoke-Command and, inside the script block, tries to read a file share on fs01. Access denied, although his account has the rights. This is the second hop problem: with the default Kerberos authentication, the remote session on web01 holds no reusable credentials for Ravi, so it cannot authenticate to a third machine on his behalf. That is a deliberate protection: a compromised web01 must not be able to use the identity of every admin who connects to it.

Ways to solve it, from preferred to last resort:

• Avoid the hop: run the command on fs01 directly, or copy the data via your own machine.
• Configure resource-based Kerberos constrained delegation, so fs01 accepts delegated credentials from web01 only (see Kerberos delegation).
• Use a Just Enough Administration (JEA) endpoint that runs under its own account with access to the share.
• Pass a credential explicitly into the block (-Credential on the inner command), keeping in mind it then exists on web01.
• CredSSP, which sends your reusable credentials to the first server: it works, but if that server is compromised, so is your account. Avoid it.

ssh has the same issue and the same kind of answer: agent forwarding (ssh -A) lets the remote host use your keys while you are connected, so when you only need to reach a host behind another one, prefer ProxyJump, which keeps the keys on your machine; when a command on the first host must reach a third, give that host its own restricted key instead.

Least privilege for remote scripts

• Run remote jobs under a dedicated account per job, not under a personal admin account; give it rights only on the machines and commands it needs (see least privilege).
• On Windows, JEA (Just Enough Administration) endpoints let a remoting user run only a listed set of commands, often as a temporary virtual account. On Linux, a restricted authorized_keys entry or a narrow sudoers rule plays that role.
• Go through a jump server where policy requires it, and keep remote commands logged (the journal and PowerShell's script block logging record what ran).
• Admin access itself should be time-limited and audited; see Privileged Access Management.

Side by side