Archives and Compression

Archiving versus compression

Two different jobs often get mixed up:

• An archive is a single file that contains many files and directories, together with their names, paths, permissions, owners and timestamps. It is easy to copy, upload or store as one unit. An archive on its own is not smaller than its contents.
• Compression rewrites data so that it takes fewer bytes, by describing repeated patterns only once. A compressor such as gzip works on one stream of bytes: it compresses a single file and knows nothing about directories.

The Unix tradition keeps the two jobs in separate tools and combines them: tar makes the archive, a compressor shrinks it. The ZIP format, common on Windows, does both in one file format.

tar: create, list and extract

tar (originally "tape archive") is the standard archiving tool on Linux and other Unix-like systems. It stores files with their permissions, owners, timestamps and symbolic links (ACLs, extended attributes and SELinux labels only with --acls, --xattrs and --selinux), so a directory restored from a tar archive looks exactly like the original. A tar archive, especially a compressed one, is called a tarball; the usual file names are .tar, .tar.gz (or .tgz), .tar.bz2 and .tar.xz. Source code and software downloads for Linux are often distributed as tarballs.

mkdir -p project/docs
printf 'Quarterly report\n' > project/docs/report.txt
printf 'item,qty\nlamp,4\n' > project/stock.csv
printf 'Project notes\n' > project/README

tar -czf project.tar.gz project
tar -tzf project.tar.gz | sort

project/
project/README
project/docs/
project/docs/report.txt
project/stock.csv

tar -czf project.tar.gz project reads as "create, gzip, file project.tar.gz, from project". tar -tzf lists what is inside. (The listing is piped through sort here because tar stores files in the order it reads them from the disk, which can differ between systems.)

-x extracts. By default into the current directory; -C names another target directory, which must already exist:

mkdir -p project/docs
printf 'Quarterly report\n' > project/docs/report.txt
tar -czf project.tar.gz project

mkdir restore
tar -xzf project.tar.gz -C restore
find restore | sort

restore
restore/project
restore/project/docs
restore/project/docs/report.txt

The options

One of -c, -t or -x is always needed; the others are optional. When reading an archive, GNU tar recognizes the compression by itself, so tar -xf project.tar.xz works without -J. When creating, you pick the compressor, or let -a pick it from the name:

mkdir -p project/docs
printf 'Quarterly report\n' > project/docs/report.txt
printf 'item,qty\nlamp,4\n' > project/stock.csv

tar -cJf project.tar.xz project
tar -cjf project.tar.bz2 project
tar -caf project.tar.gz project

for archive in project.tar.xz project.tar.bz2 project.tar.gz; do
  echo "$archive: $(tar -tf "$archive" | wc -l) entries"
done

project.tar.xz: 4 entries
project.tar.bz2: 4 entries
project.tar.gz: 4 entries

You will also see the old style without a dash, tar czf project.tar.gz project, and verbose variants such as tar -xzvf. They mean the same.

Option order: -f comes last

In a group of single-letter options, -f takes the next word as the archive name. In -czf project.tar.gz that word is project.tar.gz, as intended. In -cfz, the "next word" is the letter z:

mkdir project
printf 'item,qty\nlamp,4\n' > project/stock.csv

tar -cfz project.tar.gz project
echo "exit status $?"
ls z
tar -tf z

tar: project.tar.gz: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
exit status 2
z
project/
project/stock.csv

tar created an uncompressed archive called z and tried to add a file called project.tar.gz to it. Keep f as the last letter, directly before the archive name.

Relative paths inside archives

tar stores paths exactly as you give them. Archive a directory by a relative name (project) and the archive contains project/…, which extracts wherever you are. Given an absolute path, GNU tar removes the leading slash and says so:

$ sudo tar -czf /srv/backup/etc.tar.gz /etc
$ sudo tar -tzf /srv/backup/etc.tar.gz | head -n 3

tar: Removing leading `/' from member names
etc/
etc/hostname
etc/fstab

That is a safety feature: extracting this archive creates ./etc/… in the current directory instead of overwriting the live /etc. To store short, clean paths without the message, change directory first with -C: tar -czf /srv/backup/etc.tar.gz -C / etc. To restore a single file, extract it somewhere harmless and copy it into place after checking it.

gzip, bzip2 and xz

gzip, gunzip and zcat

gzip is the most widely used compressor on Linux: fast, supported everywhere, and good enough for most jobs. It replaces a file by a compressed version with the extension .gz; gunzip (or gzip -d) reverses that. Several tools read .gz files without unpacking them: zcat, zless and zgrep. That is how you read rotated logs such as /var/log/syslog.2.gz.

printf '09:00 INFO start\n09:05 ERROR disk full\n' > app.log
gzip app.log
ls app.log*
zcat app.log.gz
zgrep ERROR app.log.gz
gunzip app.log.gz
ls app.log*

app.log.gz
09:00 INFO start
09:05 ERROR disk full
09:05 ERROR disk full
app.log

Note that gzip app.log replaced the original. Use -k (keep) to leave it in place, -c to write to standard output instead, and -1 (fastest) to -9 (smallest) to choose the trade-off; the default is 6. gzip compresses one file at a time: gzip *.log makes one .gz per file, not an archive.

bzip2 and xz

bzip2 (.bz2) and xz (.xz) work the same way, with the same -d, -k, -c and -1…-9 options, and have their own bzcat and xzcat. They compress better than gzip at the cost of more processor time. xz usually gives the smallest files and decompresses reasonably fast, which is why distributions and the Linux kernel source use it for downloads; bzip2 is older and mostly found in existing archives.

Comparing the results

The compression ratio is the compressed size compared with the original. It depends entirely on the data: repetitive text such as logs shrinks dramatically, while photos, videos and files that are already compressed hardly shrink at all. Here the same generated web server log goes through each tool:

# 20,000 lines of a web server log, generated so the example is repeatable
seq 1 20000 | awk '{ printf "10.0.%d.%d - GET /products?id=%d %d %d ms\n",
    $1 % 7, $1 % 250, ($1 * 7919) % 10007, ($1 % 50 ? 200 : 404), ($1 * 37) % 500 }' > access.log
head -n 2 access.log

gzip  -k -n access.log      # -k keeps the original, -n leaves out name and time
bzip2 -k    access.log
xz    -k    access.log
zip   -q    access.zip access.log

original=$(stat -c %s access.log)
for file in access.log access.log.gz access.log.bz2 access.log.xz access.zip; do
  size=$(stat -c %s "$file")
  printf '%-15s %7d bytes  %3d%%\n' "$file" "$size" $(( size * 100 / original ))
done

10.0.1.1 - GET /products?id=7919 200 37 ms
10.0.2.2 - GET /products?id=5831 200 74 ms
access.log       904599 bytes  100%
access.log.gz    165506 bytes   18%
access.log.bz2    72648 bytes    8%
access.log.xz     66360 bytes    7%
access.zip       165658 bytes   18%

ZIP uses the same compression method as gzip (Deflate), hence the nearly identical size. Compressing something twice does not help; it usually adds a few bytes:

seq 1 20000 > numbers.txt
xz -k numbers.txt
echo "numbers.txt.xz:      $(stat -c %s numbers.txt.xz) bytes"
gzip -k -n numbers.txt.xz
echo "numbers.txt.xz.gz:   $(stat -c %s numbers.txt.xz.gz) bytes"

numbers.txt.xz:      4948 bytes
numbers.txt.xz.gz:   4971 bytes

Newer compressors such as zstd (.zst, tar option --zstd) combine good ratios with high speed and are increasingly used by distributions for their packages.

zip and unzip

Windows and macOS open ZIP files without extra software, so use zip when the archive is meant for people on those systems. The zip and unzip packages are not always installed on minimal servers.

mkdir -p project/docs
printf 'Quarterly report\n' > project/docs/report.txt
printf 'item,qty\nlamp,4\n' > project/stock.csv

zip -q -r project.zip project
unzip -Z1 project.zip | sort

unzip -q project.zip -d unpacked
find unpacked -type f | sort

project/
project/docs/
project/docs/report.txt
project/stock.csv
unpacked/project/docs/report.txt
unpacked/project/stock.csv

• zip -r is needed to include directories; -q silences the list of added files.
• unzip -l lists the contents with sizes and dates (-Z1, used here, prints only the names); -d sets the target directory.
• ZIP does not keep Linux owners and groups, and not every tool restores permissions or symbolic links from it. For Linux-to-Linux transfers and backups, use tar.

Extracting safely

Extracting an archive writes files. An archive from someone else decides the names of those files, so look before you extract:

1. List first with tar -tf or unzip -l. Check that everything sits under one top-level directory. An archive without one spills its files into the current directory, mixed with what is already there.
2. Extract into a new, empty directory with -C (tar) or -d (unzip), and look at the result before moving anything into place.
3. Do not extract untrusted archives as root. As root, tar restores the owners and permissions stored in the archive, including setuid bits.

GNU tar protects you from the worst cases by default: it strips a leading / and refuses member names containing .., which blocks the classic path tricks; a symbolic link already present in the target directory can still send files elsewhere, which is another reason to extract into a new, empty directory. The option -P (--absolute-names) turns that protection off; do not use it on archives you did not create yourself.

What tar does not protect you from is overwriting. It replaces existing files with the same name without asking. Here a file edited after the backup is silently rolled back, and -k (--keep-old-files) prevents that:

printf 'version 1\n' > notes.txt
tar -cf notes.tar notes.txt

printf 'version 2, edited after the backup\n' > notes.txt
tar -xf notes.tar
cat notes.txt

printf 'version 2, edited after the backup\n' > notes.txt
tar -xkf notes.tar
echo "exit status $?"
cat notes.txt

version 1
tar: notes.txt: Cannot open: File exists
tar: Exiting with failure status due to previous errors
exit status 2
version 2, edited after the backup

Simple backups with tar

tar plus a date in the file name is the simplest backup there is. Put the date in the ISO 8601 format of the International Organization for Standardization (year-month-day, date +%F) so that names sort in time order, and leave out what does not need saving with --exclude, placed before the names of what to archive:

mkdir -p site/cache site/css
printf '<h1>Shop</h1>\n' > site/index.html
printf 'body { margin: 0; }\n' > site/css/style.css
printf 'cached page\n' > site/cache/page1.html
printf 'draft\n' > site/upload.tmp

tar -czf site.tar.gz --exclude='*.tmp' --exclude='site/cache' site
tar -tzf site.tar.gz | sort

site/
site/css/
site/css/style.css
site/index.html

A backup that runs every day also needs a way to remove old copies. This example makes five dated archives, with fixed dates so that it always gives the same result, and keeps the newest three:

mkdir -p data backups
printf 'id,total\n1,19.95\n' > data/orders.csv

# In a real script: stamp=$(date +%F), which gives today's date as YYYY-MM-DD
for stamp in 2026-09-27 2026-09-28 2026-09-29 2026-09-30 2026-10-01; do
  tar -czf "backups/data-$stamp.tar.gz" data
done
ls backups

# Keep the newest three: ISO dates sort in time order
ls backups/data-*.tar.gz | sort | head -n -3 | xargs rm --
ls backups

data-2026-09-27.tar.gz
data-2026-09-28.tar.gz
data-2026-09-29.tar.gz
data-2026-09-30.tar.gz
data-2026-10-01.tar.gz
data-2026-09-29.tar.gz
data-2026-09-30.tar.gz
data-2026-10-01.tar.gz

head -n -3 prints every line except the last three, so only the older archives reach rm. On a real server such a script runs every night from cron or a systemd timer (see Scheduling Tasks):

$ sudo sh -c 'umask 077; tar -czf "/srv/backup/etc-$(date +%F).tar.gz" -C / etc'

The umask 077 keeps the archive readable by root only: it contains /etc/shadow and the SSH host keys.

A tar file on the same disk protects against mistakes, not against a broken disk, ransomware or a fire. Copy backups to another machine or medium, keep several generations, and test restores regularly; the strategy behind that is in Backup and Recovery.

Verifying archives

An archive you cannot read is not a backup. Two quick checks catch truncated or corrupted files, for example after an interrupted copy or a full disk:

• gzip -t file.gz (and xz -t, bzip2 -t) tests the compressed data and its checksum without writing anything.
• tar -tzf file.tar.gz reads the whole archive to list it; an exit status of 0 means tar could read it to the end.

mkdir project
printf 'item,qty\nlamp,4\n' > project/stock.csv
tar -czf good.tar.gz project
head -c 60 good.tar.gz > broken.tar.gz

gzip -t good.tar.gz && echo "good.tar.gz: compressed data OK"
echo "good.tar.gz: $(tar -tzf good.tar.gz | wc -l) entries listed"
gzip -t broken.tar.gz || echo "broken.tar.gz: exit status $?"

good.tar.gz: compressed data OK
good.tar.gz: 2 entries listed

gzip: broken.tar.gz: unexpected end of file
broken.tar.gz: exit status 1

broken.tar.gz holds only the first 60 bytes of the archive, as after a copy that was cut off. These checks prove the file is readable, not that it holds the right files: for that, list its contents, and now and then do a real test restore into an empty directory. tar -d (--diff) compares an archive with the files on disk and reports what has changed since.