Logs and Monitoring

Why logs matter

A server has no screen that shows error dialogs. When a service stops, a disk fails or someone tries a thousand passwords, the evidence is in the logs: timestamped lines written by the kernel, by services and by the system itself. Logs answer the questions of every incident — what happened, when, and in which order — and they are how you notice problems before users do.

Monitoring is the other half: measuring resources (disk, memory, processor load) so that "the server is slow" becomes "the root file system is 96% full" or "the load is three times the number of cores". This page shows the command-line tools for both. Larger environments send logs and measurements to central systems, but those collect exactly what these tools show. How that works — log shippers, central log stores, metrics and alerting — is the subject of Observability & SRE.

The systemd journal

The journal is the central log of a system that runs systemd, kept by the systemd-journald daemon. It collects kernel messages, messages from the boot process, everything that services write to their output, and messages sent with the syslog protocol, and stores them in an indexed binary format with extra fields such as the unit, the process ID (PID) and the priority.

Where it is stored decides whether it survives a reboot. With the default setting (Storage=auto in /etc/systemd/journald.conf) the journal is written to /var/log/journal/ if that directory exists, and otherwise only to /run/log/journal/, which is lost at every reboot. journalctl --list-boots shows how many boots are kept; if it shows only one, you cannot look at the boot before a crash.

Reading it with journalctl

journalctl is the command that reads and filters the journal. Without options it shows everything, oldest first, in a pager; the options below turn that into an answer. The unit-level basics are also on Services and systemd.

The priorities are the same eight severity levels that syslog uses (see severity levels). Options combine, so the question "what went wrong since this morning's boot?" is one command:

$ sudo journalctl -p err -b --no-pager

Sep 30 08:13:55 web01 kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PC00.PEG1.PEGP], AE_NOT_FOUND (20230628/dswload2-162)
Sep 30 10:02:17 web01 (python)[2311]: inventory-api.service: Failed at step EXEC spawning /opt/inventory/venv/bin/python: No such file or directory
Sep 30 10:02:31 web01 kernel: I/O error, dev sdb, sector 1953456 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

Each line has the same shape: timestamp, host name, the program (with its PID in brackets) and the message. Not every error matters equally: harmless firmware complaints like the first line appear on many machines at every boot. That is why it pays to know what a normal boot of your servers looks like. The last line, a read error on a disk, does matter.

Classic log files in /var/log

Before the journal, every service wrote plain text files in /var/log, many of them through a syslog daemon (today usually rsyslog). Most distributions still keep that setup next to the journal, so the same message can be in both. The file names differ between Debian and Ubuntu on one side and Red Hat Enterprise Linux (RHEL) and Fedora on the other:

Text logs are read with the tools from Viewing and Editing Text (less, tail -f) and searched with grep. Because every line has the same structure, a short pipeline turns a log into a summary. This example builds a small syslog-style file and counts the error lines per program, then the failed logins per source address:

cat > syslog <<'EOF'
Sep 30 10:00:01 web01 CRON[2240]: (www-data) CMD (php /var/www/shop/cron.php)
Sep 30 10:00:14 web01 sshd[2251]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2
Sep 30 10:00:16 web01 sshd[2251]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2
Sep 30 10:01:40 web01 sshd[2263]: Accepted publickey for ana from 192.0.2.15 port 50412 ssh2: ED25519 SHA256:q1Lx0Vc8…
Sep 30 10:02:17 web01 inventory-api[2311]: ERROR could not connect to database at 127.0.0.1:5432: Connection refused
Sep 30 10:02:22 web01 inventory-api[2318]: ERROR could not connect to database at 127.0.0.1:5432: Connection refused
Sep 30 10:02:28 web01 inventory-api[2325]: ERROR could not connect to database at 127.0.0.1:5432: Connection refused
Sep 30 10:02:31 web01 kernel: [183021.442310] I/O error, dev sdb, sector 1953456 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Sep 30 10:03:01 web01 postfix/smtp[2402]: 4F2A81C0B3: to=<orders@example.com>, relay=none, delay=30, status=deferred (connect to mail.example.com[198.51.100.25]:25: Connection timed out)
Sep 30 10:04:12 web01 sshd[2270]: Failed password for root from 198.51.100.77 port 40022 ssh2
EOF

echo "Error lines per program:"
grep -iE 'error|fail' syslog |
  awk '{ prog = $5; sub(/\[.*/, "", prog); sub(/:$/, "", prog); print prog }' |
  sort | uniq -c | sort -rn

echo "Failed SSH logins per source address:"
grep 'Failed password' syslog | awk '{ print $(NF-3) }' | sort | uniq -c

Error lines per program:
      3 sshd
      3 inventory-api
      1 kernel
Failed SSH logins per source address:
      1 198.51.100.77
      2 203.0.113.45

• grep -iE 'error|fail' keeps lines that contain "error" or "fail" in any case — a rough but useful first filter.
• In awk, $5 is the fifth field (sshd[2251]:); the two sub() calls cut off the PID and the colon, leaving the program name.
• sort | uniq -c counts identical lines (they must be adjacent, hence the sort); sort -rn puts the highest count first.
• In a "Failed password" line the source address is always the fourth field from the end, $(NF-3), whatever the user name looks like.

On a real Ubuntu server the same pipeline runs on /var/log/syslog or /var/log/auth.log. The timestamp format may differ: Ubuntu 24.04's rsyslog writes full dates such as 2026-09-30T10:00:14.512031+00:00, which is one field instead of three, so the program name moves from $5 to $3; $(NF-3) still works because it counts from the end.

Kernel messages: dmesg

dmesg prints the kernel ring buffer: the messages the kernel itself has written since boot — hardware it found, drivers it loaded, disks and USB (Universal Serial Bus) devices that appeared or failed, and processes it killed when memory ran out. The buffer has a fixed size, so the oldest messages drop off on a long-running system; the journal keeps them.

$ sudo dmesg -T | tail -n 6

[Wed Sep 30 09:41:12 2026] usb 2-1: new SuperSpeed USB device number 3 using xhci_hcd
[Wed Sep 30 09:41:12 2026] sd 6:0:0:0: [sdc] 61440000 512-byte logical blocks: (31.5 GB/29.3 GiB)
[Wed Sep 30 09:41:12 2026]  sdc: sdc1
[Wed Sep 30 10:02:31 2026] I/O error, dev sdb, sector 1953456 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[Wed Sep 30 10:27:05 2026] Out of memory: Killed process 3312 (java) total-vm:8123456kB, anon-rss:3921344kB, file-rss:0kB, shmem-rss:0kB, UID:1002 pgtables:8124kB oom_score_adj:0
[Wed Sep 30 10:27:05 2026] oom_reaper: reaped process 3312 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Without -T, timestamps are seconds since boot ([183021.442310]). Useful variations: dmesg -w follows new messages (handy while plugging in a device), dmesg -l err,warn filters by level, and journalctl -k -b -1 shows the kernel messages of the previous boot. On Debian and Ubuntu, reading the buffer needs root (dmesg: read kernel buffer failed: Operation not permitted otherwise).

Two lines in that output deserve an immediate ticket: an I/O (input/output) error on a disk (check its health and your backups) and the out-of-memory (OOM) killer, which ends a process when the system has run out of memory (see out of memory). A service that "randomly stops" is often an OOM kill.

Log rotation

Logs grow forever unless something stops them. A disk that fills up with logs stops databases, mail and logins. Log rotation solves that: at a fixed interval or size, the current log file is renamed and a fresh one started, older files are compressed, and the oldest are deleted. On most systems logrotate does this, started once a day by a systemd timer (logrotate.timer) or by cron, following the rules in /etc/logrotate.conf and one file per package in /etc/logrotate.d/:

/var/log/nginx/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    create 0640 www-data adm
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

Rotate daily and keep 14 old files; don't complain if a log is missing or rotate it when empty; compress old files, but not the most recent one (delaycompress, so a program that still writes to it briefly is not disturbed); create the new file with these permissions; and afterward tell nginx to reopen its log files. The result in the directory:

access.log         ← today, being written
access.log.1       ← yesterday, not yet compressed
access.log.2.gz    ← two days ago
access.log.3.gz
…
access.log.14.gz   ← oldest; deleted at the next rotation

Compressed logs cannot be read with cat or grep directly, but each has a z version that decompresses on the fly: zcat, zless and zgrep. zgrep also reads uncompressed files, so it can search all generations at once:

printf '%s\n' "Sep 28 03:12:40 web01 sshd[901]: Failed password for root from 198.51.100.77 port 39001 ssh2" > auth.log.2
printf '%s\n' "Sep 29 22:05:13 web01 sshd[1774]: Failed password for invalid user test from 203.0.113.45 port 50110 ssh2" \
              "Sep 29 22:05:15 web01 sshd[1774]: Failed password for invalid user test from 203.0.113.45 port 50110 ssh2" > auth.log.1
printf '%s\n' "Sep 30 10:04:12 web01 sshd[2270]: Failed password for root from 198.51.100.77 port 40022 ssh2" > auth.log
gzip auth.log.2
ls -1 auth.log*
echo "---"
zgrep -h 'Failed password' auth.log* | wc -l
zcat auth.log.2.gz | cut -d' ' -f1-3

auth.log
auth.log.1
auth.log.2.gz
---
4
Sep 28 03:12:40

-h leaves out the file names that zgrep would otherwise put in front of each match. The journal rotates itself: it keeps its total size under a limit (SystemMaxUse= in journald.conf, by default 10% of the file system, capped at 4 GB), and sudo journalctl --vacuum-time=4weeks or --vacuum-size=500M trims it by hand.

Disk usage: df and du

Two commands with confusingly similar names answer two different questions. df (disk free) asks each mounted file system how full it is. du (disk usage) adds up the sizes of files and directories you point it at. -h makes both print human-readable sizes.

$ df -h

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           392M  1.2M  391M   1% /run
/dev/sda2        48G   44G  2.1G  96% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
/dev/sda1       974M  182M  725M  21% /boot
/dev/sdb1       492G  401G   66G  86% /srv/data

Read the Use% and Mounted on columns: the root file system / is 96% full. Size minus Used is more than Avail because ext4 reserves about 5% for root, so that the system keeps working a little longer when users fill the disk. tmpfs lines are file systems in memory and can usually be ignored. A monitoring check does exactly this, with a threshold:

cat > df.txt <<'EOF'
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           392M  1.2M  391M   1% /run
/dev/sda2        48G   44G  2.1G  96% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
/dev/sda1       974M  182M  725M  21% /boot
/dev/sdb1       492G  401G   66G  86% /srv/data
EOF
awk 'NR > 1 { use = $5; sub(/%/, "", use); if (use + 0 >= 80) print $6, "is", $5, "full" }' df.txt

/ is 96% full
/srv/data is 86% full

On a live system you would pipe df -h into the same awk. use + 0 turns the text 96 into a number, so the comparison is numeric.

Out of inodes

A file system can also run out of inodes, the records that describe each file, while plenty of space is left. The symptom is "No space left on device" although df -h shows free space; the cause is usually millions of tiny files (session files, a mail queue, cache entries). df -i shows it:

$ df -i /

Filesystem      Inodes   IUsed IFree IUse% Mounted on
/dev/sda2      3145728 3145728     0  100% /

Finding what uses the space

du -sh gives one total per argument (-s, summarize). To find the culprit on a full disk, list the size of each subdirectory, sort by size, and descend into the biggest:

$ sudo du -xh --max-depth=1 /var | sort -h | tail -n 5
$ sudo du -sh /var/log/*.gz

212M    /var/cache
1.1G    /var/lib
2.3G    /var/log
14G     /var/backups-old
18G     /var
4.0K    /var/log/alternatives.log.2.gz
28K     /var/log/dpkg.log.2.gz
…

• --max-depth=1 (short: -d 1): one line per direct subdirectory plus the total.
• -x: stay on this file system, so a mounted data disk under the directory is not counted.
• sort -h understands K, M and G; a plain sort -n would put 14G before 212M.
• sudo is needed to read every directory; without it you get "Permission denied" lines and a total that is too low.

Here a forgotten backup directory is the answer. du on a large tree can take minutes and cause a lot of disk activity — on a busy server, start from the most likely directories (/var/log, /var/lib, /home, /tmp) rather than /.

Memory: free

$ free -h

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.9Gi       214Mi        12Mi       1.9Gi       1.7Gi
Swap:          2.0Gi       256Ki       2.0Gi

The free column looks alarming: only 214 MiB. It is not. Linux uses otherwise idle memory as a cache for disk contents (buff/cache) and gives it back the moment a program needs it. The number that answers "how much memory can programs still get?" is available: 1.7 GiB here. Low free is normal and healthy; low available together with growing swap use is the real warning sign. How virtual memory and swapping work is explained in Memory Management.

Load average and uptime

$ uptime
$ nproc

 10:42:07 up 42 days,  3:15,  2 users,  load average: 3.91, 2.14, 1.20
4

uptime shows the time, how long the system has run since the last boot, how many users are logged in, and three load figures. The load average is the average number of tasks that were running, waiting for a CPU (central processing unit) core, or waiting uninterruptibly for disk I/O, over the last 1, 5 and 15 minutes. It is not a percentage: compare it with the number of cores, which nproc prints. The background is explained in CPU Scheduling.

loadavg="3.91 2.14 1.20 5/214 5120"   # the contents of /proc/loadavg
cores=4                               # what nproc printed
echo "$loadavg" | awk -v cores="$cores" '{
  printf "per core:  1 min %.2f   5 min %.2f   15 min %.2f\n", $1 / cores, $2 / cores, $3 / cores
}'

per core:  1 min 0.98   5 min 0.54   15 min 0.30

Per core, around 1.0 means fully busy; well above 1.0 means work is queuing. The trend matters as much as the value: 0.30 → 0.54 → 0.98 from the 15- to the 1-minute figure means the load is rising right now; the reverse means a busy period is ending. A high load with an idle CPU points to processes waiting for disk or network storage, which is where vmstat and iostat help. To see which processes cause the load, use top or htop (see Processes and Jobs). Load is a saturation signal; how to plan with utilization and saturation is on Capacity Planning and Performance.

vmstat and iostat

vmstat prints one line of system-wide figures per interval — here every 2 seconds, 3 times. The first line is the average since boot; the following lines are live.

$ vmstat 2 3

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0    256 219140  61220 1934800    0    0    12    30  210  380  6  2 91  1  0
 0  6    256 215880  61220 1935120    0    0  8420   112 1320 2410  3  4 22 71  0
 0  5    256 214300  61224 1935400    0    0  7984    96 1288 2302  2  4 25 69  0

• r: processes running or waiting for a CPU; b: processes blocked, waiting for I/O.
• si / so: memory swapped in and out per second. Anything above zero for a longer time means memory is short.
• us, sy, id, wa: CPU time in user programs, in the kernel, idle, and waiting for I/O. The newer vmstat on Ubuntu 24.04 adds a gu column (time spent running virtual machine guests).

In the live lines, six processes are blocked (b) and the CPU spends about 70% of its time waiting for I/O (wa): the disk is the bottleneck, not the processor. iostat -xz 2 (from the sysstat package) then shows which disk: look for a device with %util near 100 and a high r_await / w_await (milliseconds per request).