The Pipeline: Text versus Objects

Two kinds of pipeline

Both shells use the same character, |, to connect commands, and in both the idea is the same: the output of one step becomes the input of the next. What travels through the connection is completely different, and that difference explains most of the other differences between Bash and PowerShell scripts.

In Bash, a pipe connects the standard output of one program to the standard input of the next. What flows through it is a text stream: a sequence of characters, usually split into lines by newline characters, with no further structure. The programs on both sides can be written in any language by anyone; the only thing they share is the agreement "lines of text". That is the strength of the Unix approach (see the Unix philosophy), and also its weakness: every program has to cut the fields it needs out of the text again.

In PowerShell, the PowerShell pipeline is an object pipeline: each cmdlet outputs .NET objects, and PowerShell hands them one by one to the next cmdlet, with all their properties and types intact. Turning objects into text happens only once, at the very end, when whatever is left is shown on screen.

The same task in both shells

Kestrel Energy's billing system exports daily meter readings as CSV (comma-separated values). Amara Okafor on the service desk wants the two meters with the highest consumption. In Bash, every step works on text: tail -n +2 skips the header line, sort is told that fields are separated by commas and that the third field is a number, head keeps two lines and cut keeps fields 1 and 3:

cat > readings.csv <<'EOF'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
M-1045,Delft,6130
EOF
tail -n +2 readings.csv | sort -t, -k3 -nr | head -n 2 | cut -d, -f1,3

M-1043,10250
M-1045,6130

The same in PowerShell. Import-Csv reads the header line and turns every other line into an object with the properties MeterId, Site and Kwh; the rest of the pipeline only names properties:

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
M-1045,Delft,6130
'@ | Set-Content readings.csv

Import-Csv readings.csv |
    Sort-Object { [int]$_.Kwh } -Descending |
    Select-Object -First 2 -Property MeterId, Kwh

MeterId Kwh
------- ---
M-1043  10250
M-1045  6130

The Bash version depends on the layout of the file: if billing adds a column in front of Kwh, -k3 and -f1,3 silently pick the wrong field. The PowerShell version depends on the names of the columns, which survive a reordering. In exchange, the Bash version works with any program that prints lines, and it is fast on files of many gigabytes. The [int] in the PowerShell version is not decoration; sorting text versus numbers explains why it is needed.

In both shells you can end a line with | and continue the pipeline on the next line, as above. That keeps long pipelines readable in scripts.

How Bash pipes text

Bash starts every command of a pipeline as a separate process, all at the same time, and connects them with pipes. The first command writes as fast as the next one reads; nothing waits for the whole file. Each program then decides on its own what a "field" is: cut and sort need a separator (-d,, -t,), awk splits on runs of spaces unless you give it -F. The filters themselves are explained in Pipes, Redirection and Filters and Text Processing in Scripts.

Because the format lives in the text, a small change in the input breaks a pipeline without an error message. Here the export gained a Date column; the script still runs, but now sorts on the date and prints dates where the kilowatt-hours should be:

cat > readings.csv <<'EOF'
MeterId,Site,Date,Kwh
M-1042,Utrecht,2026-10-05,4521
M-1043,Utrecht,2026-10-05,10250
M-1044,Delft,2026-10-05,987
EOF
# The old pipeline, written when Kwh was field 3
tail -n +2 readings.csv | sort -t, -k3 -nr | head -n 2 | cut -d, -f1,3

M-1044,2026-10-05
M-1043,2026-10-05

Defensive Bash scripts therefore look the column up by name in the header, or check the header before they start. A short awk program can do the lookup:

cat > readings.csv <<'EOF'
MeterId,Site,Date,Kwh
M-1042,Utrecht,2026-10-05,4521
M-1043,Utrecht,2026-10-05,10250
M-1044,Delft,2026-10-05,987
EOF
awk -F, 'NR == 1 { for (i = 1; i <= NF; i++) col[$i] = i; next }
         { print $col["MeterId"] "," $col["Kwh"] }' readings.csv |
  sort -t, -k2 -nr | head -n 2

M-1043,10250
M-1042,4521

One more consequence of separate processes: in Bash, every command of a pipeline runs in a subshell, a copy of the shell (unless shopt -s lastpipe is set, which runs the last command in the current shell). A variable set inside … | while read … is set in that copy and is gone when the pipeline ends:

printf '%s\n' M-1042 M-1043 M-1044 > meters.txt
count=0
cat meters.txt | while read -r meter; do
  count=$(( count + 1 ))
done
echo "Counted (pipeline):    $count"

count=0
while read -r meter; do
  count=$(( count + 1 ))
done < meters.txt
echo "Counted (redirection): $count"

Counted (pipeline):    0
Counted (redirection): 3

Reading the file with a redirection (< meters.txt) keeps the loop in the current shell. Loops covers reading files line by line safely.

How PowerShell pipes objects

A cmdlet in a PowerShell pipeline receives objects one at a time and can pass each one on before the previous cmdlet has finished, so a PowerShell pipeline also streams. A few cmdlets have to see everything first: Sort-Object cannot output the smallest item before it has read the last one, and the same holds for sort in Bash.

To find out what an object has to offer, pipe it to Get-Member. It lists the type name and every property and method:

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
'@ | Set-Content readings.csv

Import-Csv readings.csv | Get-Member

   TypeName: System.Management.Automation.PSCustomObject

Name        MemberType   Definition
----        ----------   ----------
Equals      Method       bool Equals(System.Object obj)
GetHashCode Method       int GetHashCode()
GetType     Method       type GetType()
ToString    Method       string ToString()
Kwh         NoteProperty string Kwh=4521
MeterId     NoteProperty string MeterId=M-1042
Site        NoteProperty string Site=Utrecht

Two things to notice. The properties are NoteProperty members of a PSCustomObject (see Arrays, Hash Tables and Objects), and every one of them is a string, including Kwh. A CSV file has no types; Import-Csv does not guess them. Objects from other cmdlets do have real types: Get-ChildItem gives Length as a number and LastWriteTime as a date.

Pipeline binding

How does the next cmdlet know what to do with an incoming object? Through pipeline binding: PowerShell matches each incoming object to a parameter of the receiving cmdlet. It first tries ByValue (the whole object fits a parameter that accepts pipeline input of that type), and then ByPropertyName (a property of the object has the same name as a parameter that accepts it). Test-Path accepts its -Path parameter both ways, so a list of plain strings works, and so does a CSV file with a column called Path:

New-Item -ItemType File -Path billing.log | Out-Null

# ByValue: each string becomes -Path
'billing.log', 'portal.log' | Test-Path

# ByPropertyName: the Path property becomes -Path
@'
Server,Path
web01,billing.log
web02,portal.log
'@ | Set-Content checks.csv
Import-Csv checks.csv | Test-Path

True
False
True
False

The help of a cmdlet shows which parameters accept pipeline input (Get-Help Test-Path -Parameter Path lists "Accept pipeline input: True (ByPropertyName, ByValue)"). In Bash the equivalent decision is yours: a program either reads standard input or takes arguments, and xargs turns lines of input into arguments for programs that only accept arguments.

Filtering rows

Where-Object is the PowerShell filter cmdlet: it passes on only the objects for which a condition is true. Inside the script block, $_ (also written $PSItem) is the current object. For simple conditions there is a shorter form without braces: Where-Object Site -eq 'Delft'.

Here is a trap that catches almost everyone who filters CSV data. Which meters used more than 5,000 kWh?

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
M-1045,Delft,6130
'@ | Set-Content readings.csv
$readings = Import-Csv readings.csv

'Compared as text:'
$readings | Where-Object { $_.Kwh -gt 5000 } | ForEach-Object MeterId
'Compared as numbers:'
$readings | Where-Object { [int]$_.Kwh -gt 5000 } | ForEach-Object MeterId

Compared as text:
M-1044
M-1045
Compared as numbers:
M-1043
M-1045

PowerShell converts the right side of a comparison to the type of the left side. $_.Kwh is a string, so 5000 becomes the string "5000" and the comparison is alphabetical: "987" is "greater" than "5000" because 9 comes after 5, and "10250" is smaller because 1 comes before 5. Cast the property ([int]$_.Kwh) or put the number on the left (5000 -lt $_.Kwh) and the comparison becomes numeric.

In Bash, awk does the filtering. It compares two values as numbers when both look like numbers, so the naive version happens to be right here:

cat > readings.csv <<'EOF'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
M-1045,Delft,6130
EOF
awk -F, 'NR > 1 && $3 > 5000 { print $1 }' readings.csv

M-1043
M-1045

For a filter on text rather than numbers, grep is the usual Bash choice; Where-Object with -like or -match is the PowerShell one. Filter as early as possible in both shells: every row you drop at the start is a row the later steps do not have to process. PowerShell users remember this as "filter left, format right".

Sorting text versus numbers

Text sorting compares character by character, so "10250" comes before "4521". Both shells sort text by default, and both get the same wrong answer on numbers until you say otherwise. In Bash, -n makes sort numeric:

printf '%s\n' M-1042,4521 M-1043,10250 M-1044,987 > kwh.csv
echo "Text sort:";    sort -t, -k2 kwh.csv
echo "Numeric sort:"; sort -t, -k2 -n kwh.csv

Text sort:
M-1043,10250
M-1042,4521
M-1044,987
Numeric sort:
M-1044,987
M-1042,4521
M-1043,10250

In PowerShell, Sort-Object Kwh sorts by the property's type, which for Import-Csv data is string. Sort by a script block that converts the value instead:

@'
MeterId,Kwh
M-1042,4521
M-1043,10250
M-1044,987
'@ | Set-Content kwh.csv
$readings = Import-Csv kwh.csv

'Text sort:'
$readings | Sort-Object Kwh | ForEach-Object { "$($_.MeterId) $($_.Kwh)" }
'Numeric sort:'
$readings | Sort-Object { [int]$_.Kwh } | ForEach-Object { "$($_.MeterId) $($_.Kwh)" }

Text sort:
M-1043 10250
M-1042 4521
M-1044 987
Numeric sort:
M-1044 987
M-1042 4521
M-1043 10250

Selecting and reshaping

Select-Object picks properties and rows. With -Property it returns new objects that have only the chosen properties; with -ExpandProperty it returns the bare values; -First, -Last and -Skip pick rows. Writing $readings.MeterId (member enumeration) is a short way to get the values of one property from every object in an array.

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
'@ | Set-Content readings.csv
$readings = Import-Csv readings.csv

$objects = $readings | Select-Object -Property MeterId
$values  = $readings | Select-Object -ExpandProperty MeterId
"-Property:       $($objects[0].GetType().Name) with MeterId = $($objects[0].MeterId)"
"-ExpandProperty: $($values[0].GetType().Name) '$($values[0])'"
"Member enumeration: $($readings.MeterId -join ', ')"

-Property:       PSCustomObject with MeterId = M-1042
-ExpandProperty: String 'M-1042'
Member enumeration: M-1042, M-1043

Calculated properties

A calculated property is a property that Select-Object computes while it builds the new objects. You write it as a hash table with a Name and an Expression script block. The Bash equivalent is a line of awk that prints a computed field. Both convert kWh to MWh (megawatt-hours):

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
'@ | Set-Content readings.csv

Import-Csv readings.csv |
    Select-Object MeterId, @{ Name = 'MWh'; Expression = { [math]::Round([int]$_.Kwh / 1000, 2) } }

MeterId   MWh
-------   ---
M-1042   4.52
M-1043  10.25

cat > readings.csv <<'EOF'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
EOF
awk -F, 'NR == 1 { print "MeterId MWh"; next }
         { printf "%s %.2f\n", $1, $3 / 1000 }' readings.csv

MeterId MWh
M-1042 4.52
M-1043 10.25

Measuring and grouping

Totals and counts per group are where objects save the most work. Measure-Object adds up a property; Group-Object collects objects with the same value. Measure-Object converts numeric strings to numbers by itself, so no cast is needed here; Group-Object simply collects the objects with the same Site:

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
M-1045,Delft,6130
'@ | Set-Content readings.csv
$readings = Import-Csv readings.csv

$total = ($readings | Measure-Object -Property Kwh -Sum).Sum
"Total: $total kWh"
$readings | Group-Object Site | ForEach-Object {
    $sum = ($_.Group | Measure-Object -Property Kwh -Sum).Sum
    '{0,-8} {1} meters {2,6} kWh' -f $_.Name, $_.Count, $sum
}

Total: 21888 kWh
Delft    2 meters   7117 kWh
Utrecht  2 meters  14771 kWh

In Bash, awk keeps the running totals in an array indexed by site. awk loops over such an array in no particular order, so the result goes through sort:

cat > readings.csv <<'EOF'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
M-1045,Delft,6130
EOF
awk -F, 'NR > 1 { total += $3 }
         END { print "Total: " total " kWh" }' readings.csv
awk -F, 'NR > 1 { count[$2]++; sum[$2] += $3 }
         END { for (s in sum) printf "%-8s %d meters %6d kWh\n", s, count[s], sum[s] }' readings.csv | sort

Total: 21888 kWh
Delft    2 meters   7117 kWh
Utrecht  2 meters  14771 kWh

Formatting belongs at the end

A format cmdlet (Format-Table, Format-List, Format-Wide, Format-Custom) turns objects into formatting instructions for the screen. Its output is no longer your data: it is a stream of internal formatting objects that only the display at the end of the pipeline understands. Anything after a format cmdlet, such as Export-Csv, Where-Object or a variable you work with later, gets those instructions instead of meter readings:

@'
MeterId,Kwh
M-1042,4521
M-1043,10250
'@ | Set-Content kwh.csv
$readings = Import-Csv kwh.csv

$selected  = $readings | Select-Object MeterId
$formatted = $readings | Format-Table MeterId
"Select-Object gives: $($selected[0].GetType().Name)"
"Format-Table gives:  $($formatted[0].GetType().Name)"

Select-Object gives: PSCustomObject
Format-Table gives:  FormatStartData

Use Select-Object to choose what to keep, and a format cmdlet only as the last step when a person will read the result; after it, only an Out- cmdlet such as Out-File or Out-String makes sense. Bash has no such split: column -t or printf padding changes the text itself, so you also format last there, because a later cut or awk would have to deal with the extra spaces.

Mixing text and objects

PowerShell can run native programs too. Their output arrives in the pipeline as strings, one per line, so you are back in the text world. ConvertFrom-Csv (or ConvertFrom-Json for JSON output) turns such text into objects again. This example runs the Linux head program from PowerShell; on Windows any native tool that prints CSV works the same way:

@'
MeterId,Site,Kwh
M-1042,Utrecht,4521
M-1043,Utrecht,10250
M-1044,Delft,987
'@ | Set-Content readings.csv

$lines = & /usr/bin/head -n 3 readings.csv
"From the native program: $($lines.Count) $($lines[0].GetType().Name) values"
$lines | ConvertFrom-Csv | ForEach-Object { "$($_.MeterId) at $($_.Site)" }

From the native program: 3 String values
M-1042 at Utrecht
M-1043 at Utrecht

Bash goes the other way when it needs structure: tools such as jq parse JSON and work with fields by name, which gives a Bash pipeline some of the safety of objects. JSON, CSV and APIs shows both shells working with structured data.

Side by side

Which approach fits the job

Neither model is better everywhere. Text pipelines shine for log files and other line-oriented data, for quick one-off questions at the prompt, and for gluing together programs that were never designed to cooperate. They need care when the data has structure: quoted CSV fields with commas inside, names with spaces, nested data. Object pipelines shine for structured data and for administration APIs (Active Directory, Microsoft 365, Azure, the Windows registry) that already return objects, and they make scripts easier to read months later because properties have names.

In practice you use both. A Bash script reaches for jq or Python when the data becomes structured; a PowerShell script calls native tools and converts their text output into objects. Knowing which side of the line your data is on, text or object, and converting deliberately at the boundary, prevents most of the bugs on this page.