Pipes, Redirection and Filters — Linux & Command Line

TL;DR

Every command has three standard streams: input (0), output (1) and error (2). By default all three are connected to your terminal.
> sends output to a file and empties that file first; >> appends; 2> redirects error messages; < reads input from a file.
2>&1 means "send errors to wherever output goes right now", so its position matters: cmd > file 2>&1 puts both in the file.
A pipe | feeds one command's output into the next command's input, so small filters (sort, uniq, cut, tr, wc) combine into answers.
uniq only removes adjacent duplicates: sort first.

The three standard streams

When the shell starts a program, it hands it three open channels, numbered by their file descriptor (the small number a process uses to refer to an open file or channel):

Number	Name	Short name	Default	Used for
0	Standard input	`stdin`	Keyboard	Data the program reads
1	Standard output	`stdout`	Terminal screen	The program's normal results
2	Standard error	`stderr`	Terminal screen	Error messages and warnings

Standard error is a separate output channel, stream 2, reserved for error messages and diagnostics. It exists so that errors stay visible even when you send the normal output somewhere else: if ls wrote "No such file" into the same stream as the file names, that message would silently end up in your output file. Because output and errors both go to the terminal by default, they look like one stream until you start redirecting.

Every process starts with three standard streams. Without redirection, input comes from the keyboard and both outputs go to the terminal.

You can see that the two output streams are different by sending one of them to a file. ls gets one name that exists and one that does not; the file name goes into the file, the error still reaches the screen:

touch report.txt
ls report.txt missing.txt > names.txt

ls: cannot access 'missing.txt': No such file or directory

Redirecting output to a file

Redirection is the shell connecting a command's standard stream to a file (or another stream) instead of the terminal. The shell sets it up before the command starts; the command itself does not know or care. The operators:

Operator	Meaning	Example
`> file`	Standard output to file; create it, or empty it first if it exists	`ls > files.txt`
`>> file`	Standard output appended to the end of file	`echo done >> run.log`
`2> file`	Standard error to file	`find / -name x 2> errors.txt`
`2>> file`	Standard error appended to file	`./backup.sh 2>> errors.log`
`2>&1`	Standard error to wherever standard output currently goes	`cmd > all.log 2>&1`
`&> file`	Both outputs to file (bash shorthand)	`cmd &> all.log`
`< file`	Standard input read from file	`wc -l < access.log`

> overwrites, >> appends. Watch the file grow and then shrink back to one line:

echo "backup started"  >  run.log
echo "backup finished" >> run.log
cat run.log
echo "---"
echo "new run" > run.log
cat run.log

backup started
backup finished
---
new run

Bash can protect you from accidental overwrites. After set -o noclobber, > refuses to replace an existing file; >| overrides the protection deliberately, and set +o noclobber switches it off again:

echo "version 1" > notes.txt
set -o noclobber
echo "version 2" > notes.txt

example.sh: line 3: notes.txt: cannot overwrite existing file

example.sh is the name of the script this example ran in. A script of yours shows its own name and line; at the prompt the message starts with bash:. Appending with >> still works with noclobber on.

Redirecting errors

Put the stream number in front of > to redirect that stream. 1> is the same as plain >; 2> catches the errors. Splitting them gives a clean result file and a separate error file:

touch report.txt
ls report.txt missing.txt > found.txt 2> errors.txt
echo "found:";  cat found.txt
echo "errors:"; cat errors.txt

found:
report.txt
errors:
ls: cannot access 'missing.txt': No such file or directory

2>&1 and why the order matters

2>&1 reads as "make stream 2 point to where stream 1 points now". The & tells the shell that 1 is a stream number, not a file called 1. Redirections are processed from left to right, so the two orders below do different things:

ls … > both.txt 2>&1: first stdout goes to the file, then stderr is pointed at the same place. Both end up in the file.
ls … 2>&1 > only-out.txt: first stderr is pointed at where stdout goes at that moment — the terminal — and only then is stdout moved to the file. Errors still reach the screen.

touch report.txt
ls report.txt missing.txt > both.txt 2>&1
ls report.txt missing.txt 2>&1 > only-out.txt
echo "--- both.txt"
cat both.txt
echo "--- only-out.txt"
cat only-out.txt

ls: cannot access 'missing.txt': No such file or directory
--- both.txt
ls: cannot access 'missing.txt': No such file or directory
report.txt
--- only-out.txt
report.txt

The first line of output is the error of the second ls, which went to the terminal. Bash also has &> file as a shorter way to send both streams to a file; it is not available in every shell (plain sh on Debian is dash, which does not have it), so scripts that must be portable use > file 2>&1.

/dev/null: the bit bucket

/dev/null is a special device file that throws away everything written to it and returns nothing when read. Redirect to it when you only care about the exit status or about one of the streams. A classic: searching the whole disk as a normal user produces hundreds of "Permission denied" lines; 2> /dev/null hides them so the real results stand out:

$ find / -name "sshd_config" 2> /dev/null

/etc/ssh/sshd_config

Hide errors deliberately, not by habit: a message you never see is a problem you never hear about.

Feeding input to a command

< file connects standard input to a file. Many commands accept a file name as an argument too, but the difference shows: with < the command reads an anonymous stream and does not know the file's name.

printf '%s\n' alpha bravo charlie > words.txt
wc -l words.txt
wc -l < words.txt

3 words.txt
3

The second form is handy in scripts, where you want the bare number.

Here documents and here strings

A here document is a block of text written directly in the command line or script, between <<WORD and a line containing only WORD, that the shell passes to the command as standard input. It is the usual way to create a small file or feed several lines to a command from a script. The delimiter word is free to choose; EOF is the convention.

server="web01"
cat <<EOF > motd.txt
Welcome to $server.
Maintenance window: Sunday 02:00-04:00 UTC.
EOF
cat motd.txt
cat <<'EOF'
Quoted delimiter: $server stays literal.
EOF

Welcome to web01.
Maintenance window: Sunday 02:00-04:00 UTC.
Quoted delimiter: $server stays literal.

With an unquoted delimiter the shell expands variables and command substitutions inside the text; with 'EOF' in quotes it passes the text exactly as written, which is what you want for a script or configuration snippet containing $ signs. <<-EOF additionally strips leading tab characters (not spaces), so the block can be indented inside a script.

A here string, <<< "text", is the one-line version (bash, not plain sh): the text plus a newline becomes standard input.

tr 'a-z' 'A-Z' <<< "disk almost full on web01"

DISK ALMOST FULL ON WEB01

Pipes

A pipe, written |, connects the standard output of one command directly to the standard input of the next, without a temporary file. The shell starts all commands of a pipeline at the same time; each one processes data as it arrives, so a pipeline works on files far larger than memory.

In a pipeline, stdout of each command becomes stdin of the next. Error messages bypass the pipe unless you add 2>&1 before the |.

Only standard output goes through the pipe. To send errors through as well, redirect them first: cmd 2>&1 | less (bash also accepts cmd |& less).

The Unix philosophy

Pipes are why Linux has so many small commands that each do one thing: sort only sorts, wc only counts. The idea, summarized by Doug McIlroy, one of the inventors of pipes at Bell Labs: write programs that do one thing well, write programs that work together, and use text streams as the universal interface. You rarely need a special tool for a question; you chain five general ones. Build a pipeline one stage at a time and look at the output after each | — that is how experienced administrators write them too. PowerShell pipes objects instead of text; the difference is on The Pipeline: Text versus Objects.

Filters

A filter command is a program that reads text from standard input (or from files named as arguments), transforms it, and writes the result to standard output, which makes it a natural building block in a pipeline. The ones you will use every week:

Command	What it does	Useful options
`sort`	Sorts lines	`-n` numeric, `-r` reverse, `-k2` by field 2, `-t:` field separator, `-u` unique, `-h` human sizes (2K, 1G)
`uniq`	Collapses adjacent identical lines	`-c` count, `-d` only duplicated lines, `-u` only unique lines
`cut`	Picks fields or characters from each line	`-d:` delimiter, `-f1,3` fields, `-c1-8` characters
`tr`	Translates or deletes characters	`tr 'a-z' 'A-Z'`, `-d` delete, `-s` squeeze repeats
`wc`	Counts lines, words, bytes	`-l` lines, `-w` words, `-c` bytes
`head` / `tail`	First / last lines	`-n 5`; `tail -n +2` skips the first line
`tee`	Copies input to a file and to standard output	`-a` append instead of overwrite

sort

By default sort compares text, following the rules of your locale (with LC_ALL=C, byte by byte), so 100 comes before 9. The examples ran with LC_ALL=C.UTF-8; in English locales sort ignores dots, dashes and case at first, so your order may differ. Add -n for numbers, -r to reverse, and -t plus -k to sort on a field:

printf '%s\n' "web03:9" "db01:100" "web01:10" "mail01:10" > disk-gb.txt
echo "text:";         sort disk-gb.txt
echo "by size:";      sort -t: -k2 -n disk-gb.txt
echo "largest first:"; sort -t: -k2 -nr disk-gb.txt | head -n 1

text:
db01:100
mail01:10
web01:10
web03:9
by size:
web03:9
mail01:10
web01:10
db01:100
largest first:
db01:100

-t: says fields are separated by colons, -k2 says sort on field 2 (strictly: from field 2 to the end of the line; -k2,2 limits it to field 2 only). Lines with equal keys, like the two 10s, are ordered by the whole line as a tie-breaker.

uniq

uniq compares each line only with the line directly before it. On unsorted input, duplicates that are not next to each other survive:

printf '%s\n' ana ben ana ana carla ben > logins.txt
echo "uniq alone:";  uniq logins.txt
echo "sort | uniq -c:"; sort logins.txt | uniq -c

uniq alone:
ana
ben
ana
carla
ben
sort | uniq -c:
      3 ana
      2 ben
      1 carla

sort | uniq -c is one of the most useful combinations on the command line: it answers "how many of each?". sort -u gives the distinct lines without counts.

cut and tr

cut takes columns out of structured text. Account files, CSV (comma-separated values) exports and log lines are all one delimiter away:

cat <<'EOF' > staff.csv
name,department,phone
Ana,Finance,x2210
Ben,IT,x2315
Carla,IT,x2316
EOF
cut -d, -f1,2 staff.csv
tail -n +2 staff.csv | cut -d, -f2 | sort | uniq -c

name,department
Ana,Finance
Ben,IT
Carla,IT
      1 Finance
      2 IT

cut treats every delimiter as a field boundary, so two spaces in a row mean an empty field between them. For output aligned with multiple spaces, squeeze the spaces first with tr -s ' ', or use awk '{print $1}', which splits on runs of whitespace. tr works on single characters, never on words:

echo "Hello World" | tr 'a-z' 'A-Z'
echo "web01   up    12 days" | tr -s ' ' | cut -d' ' -f2
echo "2026-10-01" | tr -d '-'
echo "a,b,c" | tr ',' '\n'

HELLO WORLD
up
20261001
a
b
c

wc, head, tail and tee

wc -l at the end of a pipeline counts results; head and tail keep the top or bottom. tee is a T-junction in the pipe: it writes its input to a file and passes it on unchanged, so you can save an intermediate result and keep going (or watch output while it is being logged):

printf '%s\n' 42 7 19 88 3 > response-ms.txt
sort -n response-ms.txt | tee sorted.txt | tail -n 2
echo "saved $(wc -l < sorted.txt) lines; fastest: $(head -n 1 sorted.txt) ms"

42
88
saved 5 lines; fastest: 3 ms

Worked example: the top 3 IP addresses in a web log

A web server is slow and you suspect one client. The question: which three IP (Internet Protocol) addresses made the most requests? The log below is in the common log format (the addresses are documentation addresses). The default format of nginx and Apache, the combined format, adds the referrer and the browser's user agent at the end of each line; the steps below work the same on it. The first step creates it; in real life it would be /var/log/nginx/access.log.

cat <<'EOF' > access.log
203.0.113.9 - - [01/Oct/2026:08:00:01 +0000] "GET / HTTP/1.1" 200 5120
198.51.100.23 - - [01/Oct/2026:08:00:03 +0000] "GET /login HTTP/1.1" 200 1830
203.0.113.9 - - [01/Oct/2026:08:00:04 +0000] "GET /style.css HTTP/1.1" 200 740
192.0.2.44 - - [01/Oct/2026:08:00:09 +0000] "POST /login HTTP/1.1" 401 210
192.0.2.44 - - [01/Oct/2026:08:00:10 +0000] "POST /login HTTP/1.1" 401 210
203.0.113.9 - - [01/Oct/2026:08:00:12 +0000] "GET /about HTTP/1.1" 200 3310
192.0.2.44 - - [01/Oct/2026:08:00:12 +0000] "POST /login HTTP/1.1" 401 210
198.51.100.7 - - [01/Oct/2026:08:00:15 +0000] "GET / HTTP/1.1" 200 5120
192.0.2.44 - - [01/Oct/2026:08:00:16 +0000] "POST /login HTTP/1.1" 401 210
198.51.100.23 - - [01/Oct/2026:08:00:20 +0000] "GET /account HTTP/1.1" 200 2290
EOF
wc -l access.log

10 access.log

Step 1 — take the first field. Fields are separated by spaces and the IP address is field 1:

$ cut -d' ' -f1 access.log | head -n 4

203.0.113.9
198.51.100.23
203.0.113.9
192.0.2.44

Step 2 — count each address. Sort, so equal addresses are adjacent, then count with uniq -c:

$ cut -d' ' -f1 access.log | sort | uniq -c

      4 192.0.2.44
      2 198.51.100.23
      1 198.51.100.7
      3 203.0.113.9

Step 3 — rank and keep three. Sort the counts numerically, largest first, and keep the top three. The complete pipeline, run against the same log:

cat <<'EOF' > access.log
203.0.113.9 - - [01/Oct/2026:08:00:01 +0000] "GET / HTTP/1.1" 200 5120
198.51.100.23 - - [01/Oct/2026:08:00:03 +0000] "GET /login HTTP/1.1" 200 1830
203.0.113.9 - - [01/Oct/2026:08:00:04 +0000] "GET /style.css HTTP/1.1" 200 740
192.0.2.44 - - [01/Oct/2026:08:00:09 +0000] "POST /login HTTP/1.1" 401 210
192.0.2.44 - - [01/Oct/2026:08:00:10 +0000] "POST /login HTTP/1.1" 401 210
203.0.113.9 - - [01/Oct/2026:08:00:12 +0000] "GET /about HTTP/1.1" 200 3310
192.0.2.44 - - [01/Oct/2026:08:00:12 +0000] "POST /login HTTP/1.1" 401 210
198.51.100.7 - - [01/Oct/2026:08:00:15 +0000] "GET / HTTP/1.1" 200 5120
192.0.2.44 - - [01/Oct/2026:08:00:16 +0000] "POST /login HTTP/1.1" 401 210
198.51.100.23 - - [01/Oct/2026:08:00:20 +0000] "GET /account HTTP/1.1" 200 2290
EOF
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 3

      4 192.0.2.44
      3 203.0.113.9
      2 198.51.100.23

Step 4 — ask the next question. The top client made four POST /login requests that all got status 401 (Unauthorized): that looks like password guessing, not a busy user. The same toolkit counts status codes. Split on the double quote, and the status code is the second word of the third piece:

$ cut -d'"' -f3 access.log | cut -d' ' -f2 | sort | uniq -c

      6 200
      4 401

Nothing here is specific to web logs. "Extract a field, sort, count, sort by count, take the top" answers questions about SSH (Secure Shell) login attempts, email senders, firewall drops and disk usage alike. For logs that live in the systemd journal, see Logs and Monitoring; for picking lines by pattern instead of by position, see grep, find and regular expressions.