Storage in Practice

The concepts behind this page (disks, partitions, partition tables, volumes and file systems) are explained in Disks, Partitions and Volumes and File Systems. This page is about doing the work on a Linux server. Most commands here change the system and need root rights, so they are shown as illustrative terminal sessions with sudo; the small examples that only process text can be run anywhere.

Block devices and their names

A block device is a device that the kernel reads and writes in fixed-size blocks and that allows random access, such as a disk, a partition, a logical volume or a USB stick; it appears as a file under /dev. File systems live on block devices. (A character device, such as a terminal, is read as a stream instead.)

The name tells you what kind of hardware is behind it:

lsblk and blkid

lsblk (list block devices) shows every block device as a tree: disks, the partitions on them, and what sits on top. It needs no root rights.

$ lsblk

NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0   40G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    2G  0 part /boot
└─sda3                      8:3    0   38G  0 part
  └─ubuntu--vg-ubuntu--lv 252:0    0   19G  0 lvm  /
sdb                         8:16   0   20G  0 disk
sr0                        11:0    1 1024M  0 rom

This is a typical Ubuntu Server virtual machine: the first disk has a tiny boot-loader partition, a /boot partition and a large partition that holds a Logical Volume Manager (LVM) logical volume with the root file system. sdb is a new, empty disk. RM marks removable devices, RO read-only ones.

lsblk -f replaces the size columns with file system information, which is usually what you want:

$ lsblk -f

NAME                      FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1
├─sda2                    ext4        1.0            9c2d41f7-5e0a-4b3c-8d61-7a2f0e9b1c44    1.6G     9% /boot
└─sda3                    LVM2_member LVM2 001       Xk3p2c-4TQe-mL8v-Rz1d-Hq7w-Bn5s-Ja0fYc
  └─ubuntu--vg-ubuntu--lv ext4        1.0            6b1e4a2c-93f0-4d7e-b1a5-0c2d9e8f7a31   12.9G    27% /
sdb
sr0

blkid (block ID) prints the same identifiers for one device or all of them. With -s UUID -o value it prints only the UUID, which is handy in scripts:

$ sudo blkid /dev/sda2
$ sudo blkid -s UUID -o value /dev/sda2

/dev/sda2: UUID="9c2d41f7-5e0a-4b3c-8d61-7a2f0e9b1c44" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="1d7e3b90-6a24-4c8f-b35e-0f9a2c7d8e61"
9c2d41f7-5e0a-4b3c-8d61-7a2f0e9b1c44

UUIDs instead of device names

A UUID (universally unique identifier) is a 128-bit number, written as 32 hexadecimal digits in five groups, that mkfs writes into a new file system so the file system can be recognized wherever the disk ends up. Device names do not have that property: sda and sdb are handed out in the order the kernel detects the disks, and that order can change when a disk is added, removed or moved to another controller. A configuration that says /dev/sdb1 may one day point at the wrong disk; one that says UUID=3f6c… is not affected by detection order. (A cloned disk carries the same UUIDs as the original, so the UUID is only unique as long as nobody copies the disk.)

Keep the two UUIDs in the blkid output apart: UUID belongs to the file system (and changes when you reformat), PARTUUID belongs to the partition entry in the GUID Partition Table (GPT). Labels (LABEL=data) are readable but not guaranteed to be unique. The same identifiers are also available as symbolic links in /dev/disk/by-uuid/, /dev/disk/by-label/ and /dev/disk/by-id/.

Picking the UUID and type out of a blkid line and turning them into an fstab line is a small text job. Note the space before UUID= in the pattern, so that PARTUUID= does not match:

# One line of blkid output for a freshly formatted partition
line='/dev/sdb1: LABEL="data" UUID="3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="data" PARTUUID="8e1d5b7a-2c4f-4a3e-9b6d-1f0e2d3c4b5a"'

uuid=$(grep -o ' UUID="[^"]*"' <<< "$line" | cut -d'"' -f2)
fstype=$(grep -o ' TYPE="[^"]*"' <<< "$line" | cut -d'"' -f2)

printf 'UUID=%s  /srv/data  %s  defaults,nofail  0  2\n' "$uuid" "$fstype"

UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42  /srv/data  ext4  defaults,nofail  0  2

Mounting and unmounting

Mounting attaches a file system to a directory in the tree, its mount point; unmounting detaches it again. The directory must exist, and whatever was in it stays hidden (not deleted) while something is mounted on top.

$ sudo mkdir -p /mnt/usb
$ sudo mount /dev/sdc1 /mnt/usb              # type detected automatically
$ sudo mount -o ro /dev/sdc1 /mnt/usb        # read-only, for inspecting evidence or a damaged disk
$ sudo mount -t xfs /dev/sdd1 /srv/archive   # name the type explicitly
$ findmnt /mnt/usb                           # is it mounted, and how?
$ sudo umount /mnt/usb                       # umount, without the first "n"

TARGET   SOURCE    FSTYPE OPTIONS
/mnt/usb /dev/sdc1 vfat   rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro

mount without arguments lists every mounted file system, including dozens of virtual ones (proc, sysfs, tmpfs, cgroup2). findmnt shows the same as a tree and is easier to read; df -h shows only real storage with its free space. You can unmount by device or by mount point.

"target is busy"

A file system cannot be unmounted while a process uses it: an open file, or a shell whose current directory is inside it. Very often that shell is your own.

$ sudo umount /srv/data
$ sudo fuser -vm /srv/data
$ sudo lsof /srv/data

umount: /srv/data: target is busy.
                     USER        PID ACCESS COMMAND
/srv/data:           root     kernel mount /srv/data
                     ana        2143 ..c.. bash
                     postgres   1807 F.... postgres
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash     2143      ana  cwd    DIR  8,17     4096      2 /srv/data
postgres 1807 postgres    5u   REG  8,17  8192000 131074 /srv/data/pg/base/16384/2619

fuser -vm lists every process using the file system (c = current directory, F = open file); lsof (list open files) shows which files. Here Ana's shell sits in /srv/data (she should cd elsewhere) and PostgreSQL has database files open (stop the service cleanly with systemctl stop). Then unmount again.

/etc/fstab

A mount made with mount is gone after a reboot. /etc/fstab (file system table) is the configuration file that lists the file systems to mount at boot, one per line, with where to mount them and with which options; systemd turns each line into a mount unit when the system starts. Here is a typical one:

# <file system>                            <mount point> <type> <options>          <dump> <pass>
UUID=6b1e4a2c-93f0-4d7e-b1a5-0c2d9e8f7a31  /              ext4   errors=remount-ro  0      1
UUID=9c2d41f7-5e0a-4b3c-8d61-7a2f0e9b1c44  /boot          ext4   defaults           0      1
UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42  /srv/data      ext4   defaults,nofail    0      2
/swap.img                                  none           swap   sw                 0      0

A few options are worth knowing by name:

• defaults means read-write, allow devices and programs, mount automatically at boot (rw,suid,dev,exec,auto,nouser,async).
• nofail: if the device is missing, continue booting instead of stopping. Use it for every disk the system can start without: data disks, USB backup disks.
• noexec,nosuid,nodev: forbid running programs, setuid bits and device files on that file system; a common hardening for /tmp or upload directories.
• _netdev: the file system needs the network (NFS, iSCSI, Internet Small Computer Systems Interface), so mount it only once the network is up.

XFS checks itself when it mounts and has its own repair tool, so its pass field is 0; ext4 uses 1 or 2. Ubuntu's installer gives /boot a 1 as well, as in the example above.

Test before you reboot

$ sudo cp /etc/fstab /etc/fstab.bak     # a way back
$ sudoedit /etc/fstab                   # make the change
$ sudo findmnt --verify                 # syntax, devices, types
$ sudo systemctl daemon-reload          # let systemd read the new fstab
$ sudo mount -a                         # mount everything not yet mounted

/srv/data
   [E] unreachable on boot required source: UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d24

0 parse errors, 1 error, 0 warnings
mount: /srv/data: can't find UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d24.

mount -a mounts every entry of fstab that is not already mounted (except those marked noauto), exactly as the boot would. If it prints nothing and findmnt /srv/data shows the mount, the entry works. If it complains, fix the line or restore the backup before you reboot. (Recent versions of mount also remind you to run systemctl daemon-reload when fstab is newer than what systemd has loaded.) findmnt --verify reports a missing device even when the entry has nofail; the boot would carry on, but the error is still worth reading.

You can catch the most common fstab mistakes with a few lines of awk. This check skips comments, prints the essentials of each entry, and flags device names and data disks without nofail:

cat > fstab <<'EOF'
# <device>                                 <dir>      <type> <options>          <dump> <pass>
UUID=6b1e4a2c-93f0-4d7e-b1a5-0c2d9e8f7a31  /          ext4   errors=remount-ro  0      1
UUID=0A1B-2C3D                             /boot/efi  vfat   umask=0077         0      1
/dev/sdb1                                  /srv/data  ext4   defaults           0      2
UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42  /backup    ext4   defaults           0      2
/swap.img                                  none       swap   sw                 0      0
EOF

awk '
  /^[[:space:]]*#/ || NF == 0 { next }        # skip comments and empty lines
  {
    note = ""
    if ($1 ~ /^\/dev\/sd/)
      note = "device name can change: use UUID="
    else if ($2 != "/" && $2 != "/boot/efi" && $3 != "swap" && $4 !~ /nofail/)
      note = "no nofail: a missing disk stops the boot"
    printf "%-10s %-5s pass=%s  %s\n", $2, $3, $6, note
  }
' fstab

/          ext4  pass=1
/boot/efi  vfat  pass=1
/srv/data  ext4  pass=2  device name can change: use UUID=
/backup    ext4  pass=2  no nofail: a missing disk stops the boot
none       swap  pass=0

Whether a missing disk should stop the boot is a judgment call: for a database volume without which the server is useless, failing loudly can be the better choice. But decide it on purpose.

Adding a new disk, end to end

The whole job for a new, empty 20 GiB data disk that appeared as /dev/sdb in the lsblk output above: one partition, an ext4 file system, a mount point and an fstab entry.

$ lsblk -f /dev/sdb                                        # 1. really empty?
$ sudo parted --script /dev/sdb mklabel gpt mkpart data ext4 0% 100%   # 2. GPT, one partition
$ sudo mkfs.ext4 -L data /dev/sdb1                          # 3. file system
$ sudo mkdir -p /srv/data                                   # 4. mount point
$ sudo blkid -s UUID -o value /dev/sdb1                     # 5. its UUID

mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 5242368 4k blocks and 1310720 inodes
Filesystem UUID: 3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42

Step 2 uses parted in script mode: a GPT and one partition from the first to the last usable megabyte, which parted aligns correctly by itself. The interactive alternative is sudo fdisk /dev/sdb (g for a new GPT, n for a new partition with the defaults, w to write). Formatting with -L data also gives the file system a label. Then add the fstab line, test it and check the result:

$ sudo cp /etc/fstab /etc/fstab.bak
$ echo 'UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42 /srv/data ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab
$ sudo systemctl daemon-reload
$ sudo mount -a
$ df -h /srv/data

UUID=3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d42 /srv/data ext4 defaults,nofail 0 2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        20G   24K   19G   1% /srv/data

sudo tee -a appends as root; a plain sudo echo … >> /etc/fstab fails because the redirection is done by your own, unprivileged shell. Note -a: without it, tee overwrites the whole file. The new file system belongs to root; give the right group access with chown and chmod (see Users, Groups and Permissions in Practice).

Why is 20 GiB shown as 20G in size but only 19G available? ext4 reserves 5% of the blocks for root by default, so that system services can still write when users have filled the disk. On a pure data disk you can lower that with sudo tune2fs -m 1 /dev/sdb1.

LVM: physical volumes, volume groups, logical volumes

A partition has a fixed size and lives on one disk. The Logical Volume Manager (LVM) is a Linux storage layer that pools disks or partitions into a volume group and cuts flexible logical volumes out of it, which can be grown, moved to other disks or snapshotted while in use. Many server installers (Ubuntu Server, RHEL) put the root file system on LVM by default. There are three layers:

• A physical volume (PV) is a disk or partition prepared for LVM with pvcreate.
• A volume group (VG) pools one or more PVs into a single store of equal-sized extents, 4 MiB each by default.
• A logical volume (LV) is a range of extents from the VG that behaves like a partition: you put a file system on it and mount it. It appears as /dev/vg/lv and /dev/mapper/vg-lv.

Creating and inspecting LVM

Each layer has the same set of verbs: create, display (details), s (a short summary), extend and remove. Building a VG called data-vg from two new disks (two further new, empty disks, /dev/sdc and /dev/sdd, checked with lsblk -f) and cutting a 30 GiB LV from it:

$ sudo pvcreate /dev/sdc /dev/sdd
$ sudo vgcreate data-vg /dev/sdc /dev/sdd
$ sudo lvcreate -n data-lv -L 30G data-vg      # -L size, or -l 100%FREE for everything
$ sudo mkfs.xfs /dev/data-vg/data-lv
$ sudo pvs; sudo vgs; sudo lvs

  PV         VG      Fmt  Attr PSize   PFree
  /dev/sdc   data-vg lvm2 a--  <20.00g      0
  /dev/sdd   data-vg lvm2 a--  <20.00g <9.99g
  VG      #PV #LV #SN Attr   VSize  VFree
  data-vg   2   1   0 wz--n- <39.99g <9.99g
  LV      VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data-lv data-vg -wi-a----- 30.00g

The <20.00g is LVM being honest: a little of each PV holds LVM's own metadata, so a 20 GiB disk offers slightly less than 20 GiB of extents. Sizes in LVM are counted in extents; the arithmetic is simple:

pe_mib=4                                  # extent size: 4 MiB (the default)
for disk_gib in 38 20; do
  extents=$(( (disk_gib * 1024 - 1) / pe_mib ))   # 1 MiB holds LVM metadata; rounds down
  echo "PV of ${disk_gib} GiB -> ${extents} extents"
done
lv_extents=$(( 29 * 1024 / pe_mib ))
echo "LV of 29 GiB needs ${lv_extents} extents"

PV of 38 GiB -> 9727 extents
PV of 20 GiB -> 5119 extents
LV of 29 GiB needs 7424 extents

Growing a volume and its file system

"The root file system is full, and the VM administrator gave us more disk" is one of the most common storage tickets. Growing takes two steps: first the block device grows, then the file system on it. A filesystem resize is the second step: changing the size of an existing file system so it uses all (or less) of the space of its block device, with resize2fs for ext4 or xfs_growfs for XFS. Skip it and lsblk shows the bigger volume while df still shows the old size.

On Ubuntu Server, the installer often leaves part of the volume group unused, so first check VFree with sudo vgs. If there is free space, one command grows the LV and, thanks to -r (--resizefs), the file system on it:

$ sudo lvextend -r -L +10G ubuntu-vg/ubuntu-lv

  Size of logical volume ubuntu-vg/ubuntu-lv changed from 19.00 GiB (4864 extents) to 29.00 GiB (7424 extents).
  Logical volume ubuntu-vg/ubuntu-lv successfully resized.
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 4
The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 7602176 (4k) blocks long.

-L +10G adds 10 GiB; -L 29G sets an absolute size; -l +100%FREE takes everything that is left. If the VG itself is full, add a disk first: sudo pvcreate /dev/sdc and sudo vgextend ubuntu-vg /dev/sdc. When a virtual disk was enlarged instead, the partition that holds the PV must grow first (sudo growpart /dev/sda 3, from the cloud-guest-utils or cloud-utils-growpart package), then the PV (sudo pvresize /dev/sda3), then the LV.

Without -r, or on a plain partition without LVM, grow the file system yourself. Both ext4 and XFS can grow while mounted:

When a disk fills up

A full file system causes strange symptoms: services that fail to start, logins that hang, databases that stop, editors that cannot save. Error messages say No space left on device. Work through it in this order.

1. Which file system, and space or inodes?

$ df -h
$ df -i /var

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.2M  391M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   19G   19G     0 100% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
/dev/sda2                          2.0G  183M  1.7G  10% /boot
/dev/sdb1                           20G  2.1G   17G  12% /srv/data
Filesystem                         Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 1245184 318402  926782   26% /

df -h (disk free, human-readable) shows the root file system at 100%. df -i shows inodes: every file needs one, and a file system can run out of them while there is still space, typically when a program creates millions of tiny files (session files, mail queues). Then df -h looks fine but df -i shows 100%. Here inodes are fine; space is the problem.

2. Where is the space?

$ sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -5
$ sudo du -xh --max-depth=1 /var | sort -h | tail -5

1.1G    /home
2.9G    /usr
13G     /var
18G     /
28K     /var/tmp
410M    /var/cache
1.2G    /var/lib
11G     /var/log
13G     /var

du (disk usage) adds up what directories contain. -x stays on one file system (so /srv/data and virtual file systems are skipped), --max-depth=1 gives one line per subdirectory, and sort -h sorts human-readable sizes correctly. Repeat one level deeper each time until you find the culprit. To list the biggest individual files, find is more direct:

mkdir -p var/log/nginx var/backups home/ana/Downloads
head -c 1200000  /dev/zero > var/log/nginx/access.log
head -c 5300000  /dev/zero > var/log/nginx/access.log.1
head -c 26000000 /dev/zero > var/backups/db-2026-09-30.sql
head -c 900000   /dev/zero > home/ana/Downloads/manual.pdf
head -c 3100000  /dev/zero > home/ana/Downloads/ubuntu-notes.tar

# Files over 1 MiB, largest first, sizes in human-readable units
find . -type f -size +1M -printf '%s\t%p\n' | sort -rn | numfmt --to=iec --field=1

25M ./var/backups/db-2026-09-30.sql
5.1M ./var/log/nginx/access.log.1
3.0M ./home/ana/Downloads/ubuntu-notes.tar
1.2M ./var/log/nginx/access.log

On a real server, run it as sudo find / -xdev -type f -size +500M …; -xdev keeps find on one file system, like du -x.

3. Deleted, but still open

A classic: someone deletes a 10 GiB log file, but df still shows the disk as full. On Linux, deleting a file removes its name; the space is freed only when the last process that has the file open closes it. A service that keeps writing to its deleted log holds on to the space.

$ sudo lsof +L1

COMMAND  PID     USER   FD   TYPE DEVICE    SIZE/OFF NLINK   NODE NAME
java    1422  appuser    4w   REG  252,0 10737418240     0 393231 /var/log/app/app.log (deleted)

+L1 lists open files with fewer than one link, that is: deleted. Restart the service (sudo systemctl restart app) and the space comes back. If a restart is not possible right now, the file can be emptied through the process: sudo truncate -s 0 /proc/1422/fd/4. To empty a log that is in use without this trouble, truncate it instead of deleting it: sudo truncate -s 0 /var/log/app/app.log.

4. Free space safely

• The journal: journalctl --disk-usage shows its size; sudo journalctl --vacuum-size=500M or --vacuum-time=2weeks removes the oldest archived journal files. Set a permanent limit with SystemMaxUse= in /etc/systemd/journald.conf (see Logs and Monitoring).
• Rotated logs (*.1, *.gz in /var/log) can be removed; the current log should be truncated, not deleted.
• Package caches: sudo apt clean or sudo dnf clean all; old kernels with sudo apt autoremove.
• Things that are not yours: database files, anything under /var/lib, container images. Ask the owner, and use the application's own tools (for containers, docker system prune after checking what it removes).

Swap files

Swap space lets the kernel move memory pages that are rarely used to disk (see Memory Management). Ubuntu's installer creates a swap file (/swap.img); RHEL and Debian create a swap logical volume or partition. A file is easier to resize. Adding a 2 GiB swap file:

$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile           # swap may contain passwords and keys
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
$ swapon --show; free -h

NAME      TYPE SIZE USED PRIO
/swapfile file   2G   0B   -2
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.1Gi       1.9Gi        12Mi       1.0Gi       2.7Gi
Swap:          2.0Gi          0B       2.0Gi

fallocate works on ext4 and XFS. On file systems where it does not produce a usable swap file (Btrfs needs its own procedure), mkswap or swapon reports an error; follow the file system's documentation then. swapoff /swapfile takes a swap file out of use again.