Troubleshooting Linux

A method

Troubleshooting Linux follows the same discipline as troubleshooting a network (see the structured methodology). On a server it comes down to five habits:

1. Define the problem precisely. "The website is down" becomes "since 09:14, nginx on web1 does not start; the other servers are fine". Who is affected, since when, and what exactly is the error message?
2. Find out what changed. Systems rarely break by themselves. An update, a configuration change, a full disk, a new firewall rule, an expired certificate: the change is usually the cause.
3. Read the logs. Linux programs say why they fail, in the journal or in /var/log. Read the actual message before forming a theory.
4. Test one thing at a time. Change one thing, test, and undo it if it did not help. Make a backup copy of every file before you edit it.
5. Document. The cause, the fix, and the commands you ran, in the ticket. The next person (often you) will thank you.

Finding what changed

Package logs show updates (/var/log/apt/history.log on Debian and Ubuntu, dnf history on RHEL), and last shows who logged in. For configuration files, find can list what was modified after a given moment. Here on a small copy of /etc with fixed dates:

mkdir -p etc/nginx/sites-enabled etc/logrotate.d etc/cron.d
touch -d '2026-06-12 10:00' etc/logrotate.d/nginx etc/cron.d/backup
touch -d '2026-09-02 14:30' etc/nginx/nginx.conf
touch -d '2026-09-29 16:45' etc/nginx/sites-enabled/shop
touch -d '2026-09-29 17:05' etc/cron.d/cleanup

# Which files changed since yesterday morning?
find etc -type f -newermt '2026-09-29 08:00' -printf '%TY-%Tm-%Td %TH:%TM  %p\n' | sort

2026-09-29 16:45  etc/nginx/sites-enabled/shop
2026-09-29 17:05  etc/cron.d/cleanup

On a real server: sudo find /etc -newermt '-1 day' -type f (files changed in the last 24 hours), or -mmin -60 for the last hour. If /etc is kept in Git (for example with etckeeper), git log answers the question even better.

First commands for any incident

Before diving into one theory, take one minute for an overview. These five commands catch a surprising number of problems immediately:

$ uptime
$ df -h
$ free -h
$ systemctl --failed
$ journalctl -p err -b

 09:16:40 up 1 day,  1:18,  1 user,  load average: 0.21, 0.30, 0.27
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   19G  8.4G  9.6G  47% /
/dev/sda2                          2.0G  183M  1.7G  10% /boot
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.2Gi       1.4Gi        12Mi       1.4Gi       2.6Gi
Swap:          2.0Gi          0B       2.0Gi
  UNIT          LOAD   ACTIVE SUB    DESCRIPTION
● nginx.service loaded failed failed A high performance web server and a reverse proxy server

1 loaded units listed.
Sep 30 09:14:02 web1 nginx[2310]: 2026/09/30 09:14:02 [emerg] 2310#2310: unexpected ";" in /etc/nginx/sites-enabled/shop:14

• uptime: how long since the last reboot (an unexpected reboot is a clue in itself) and the load average (see a slow system).
• df -h: is any file system full? A full disk causes failures that look unrelated.
• free -h: is memory exhausted and swap in heavy use? Look at available, not free.
• systemctl --failed: which units have failed.
• journalctl -p err -b: every message of priority "error" or worse since the last boot. Add -e to jump to the end.

Here the answer is already visible: a syntax error on line 14 of an nginx site file that was changed yesterday afternoon. Two other directions the overview can point to:

A full disk

df -h at 100% (or df -i at 100% inodes) explains failing services, logins that hang and applications that cannot save. How to find what fills it, including deleted files that are still open, and how to free space safely is described in Storage in Practice: when a disk fills up.

Network trouble

Work up from the host: does the interface have an address and a default route (ip addr, ip route), does the gateway answer (ping), do names resolve (dig, resolvectl status), is the service listening (ss -tlnp), and does the host firewall let it through? The commands are covered in Networking from the Command Line, and the layer-by-layer approach in Troubleshooting Tools and Method.

A service won't start

Start with the status, which shows the state, the exit code and the last log lines of the unit (see reading systemctl status):

$ systemctl status nginx

× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2026-09-30 09:14:02 UTC; 2min 38s ago
       Docs: man:nginx(8)
    Process: 2310 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 18ms

Sep 30 09:14:02 web1 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Sep 30 09:14:02 web1 nginx[2310]: 2026/09/30 09:14:02 [emerg] 2310#2310: unexpected ";" in /etc/nginx/sites-enabled/shop:14
Sep 30 09:14:02 web1 nginx[2310]: nginx: configuration file /etc/nginx/nginx.conf test failed
Sep 30 09:14:02 web1 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Sep 30 09:14:02 web1 systemd[1]: nginx.service: Failed with result 'exit-code'.
Sep 30 09:14:02 web1 systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.

The status shows only the last ten lines. When the cause is not among them, read the unit's full log: journalctl -u nginx -b (this boot), or journalctl -xeu nginx (jump to the end, with extra explanations). Look for the first error, not the last: later messages are often consequences.

Test the configuration

Most daemons can check their configuration without starting. Run the test after every change and before every restart, so a typo never takes a running service down:

$ sudo sshd -t

/etc/ssh/sshd_config line 42: Bad configuration option: PermitRootLogn
/etc/ssh/sshd_config: terminating, 1 bad configuration options

Port already in use

A service that cannot open its network port fails with a message like bind() to 0.0.0.0:80 failed (98: Address already in use). Something else listens on that port already; ss tells you what:

$ sudo ss -tlnp 'sport = :80'

State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("apache2",pid=1432,fd=4),("apache2",pid=1431,fd=4),("apache2",pid=1430,fd=4))

Apache was installed as a dependency and started first. Decide which of the two should serve port 80, then stop and disable the other (sudo systemctl disable --now apache2) or move one of them to another port. -p needs sudo to show processes of other users.

When a program crashes: core dumps

If the status shows code=dumped, status=11/SEGV or status=6/ABRT, the program did not exit with an error but crashed. A core dump is a file containing the memory of a process at the moment it crashed, which developers load into a debugger to see where and why it failed. On systemd-based distributions, systemd-coredump collects them:

$ coredumpctl list
$ coredumpctl info 3120

TIME                           PID  UID  GID SIG     COREFILE EXE                    SIZE
Wed 2026-09-30 14:02:11 UTC   3120 1001 1001 SIGSEGV present  /usr/local/bin/invoicer 1.2M

You rarely analyze a core dump yourself; you attach it (or the output of coredumpctl info) to a bug report for the vendor. Ubuntu may collect crashes with its own tool, Apport, in /var/crash instead. Treat core dumps as sensitive: they contain whatever the program had in memory, including passwords. Shells usually start with core files disabled (ulimit -c prints 0).

"Permission denied"

Four questions, in this order, find almost every cause:

1. Who is asking? Your own shell (id shows user and groups; new group memberships only count after you log in again), or a service (ps -o user= -C nginx)?
2. What does the file allow? ls -l file. A + after the permission bits means there are extra access control list (ACL) entries: getfacl file.
3. Can that user get there? Every directory on the path needs execute (x) permission for that user. namei -l /full/path shows each step.
4. Is something else in the way? SELinux or AppArmor, a file system mounted read-only or with noexec (findmnt -T /path), or the immutable attribute (lsattr file shows an i; that gives "Operation not permitted", even for root).

The third question is the one people skip. The file below is readable for everyone, and yet only its owner can read it, because one directory on the way is closed:

umask 022
mkdir -p srv/reports/2026
printf 'Q3 revenue: 1.2M\n' > srv/reports/2026/q3.txt
chmod 700 srv/reports                  # someone "tightened" one directory

namei -m srv/reports/2026/q3.txt

f: srv/reports/2026/q3.txt
 drwxr-xr-x srv
 drwx------ reports
 drwxr-xr-x 2026
 -rw-r--r-- q3.txt

namei -m shows the mode of every component; namei -l also shows owner and group, which you need to decide which permission set applies. reports is drwx------: other users cannot enter it, so everything below is out of reach. The fix is to open that one directory to the right group (chmod 750 plus the right group owner), not chmod -R 777 on the tree.

If permissions and path are fine on a RHEL-family system, check for SELinux denials with sudo ausearch -m AVC -ts recent; on Ubuntu, look for apparmor="DENIED" in journalctl -k. Both are explained in Linux Security Basics.

"command not found"

The shell searches for a command in the directories listed in the PATH environment variable, in order, and stops at the first match. "command not found" has three usual causes: a typo, a program that is not installed, or a program that is installed in a directory that is not in PATH. command -v and type show what the shell would run:

PATH=/usr/local/bin:/usr/bin:/bin       # start from a typical PATH
# A script installed in ~/bin, a directory that is not (yet) in PATH
mkdir -p ~/bin
printf '#!/bin/sh\necho "Nightly backup finished"\n' > ~/bin/nightly-backup
chmod +x ~/bin/nightly-backup

command -v nightly-backup || echo "nightly-backup: not found in PATH"
echo "$PATH" | tr ':' '\n'

PATH="$HOME/bin:$PATH"                 # add it, for this shell only
command -v nightly-backup | sed "s|^$HOME|~|"
nightly-backup

type cd                                # not a file at all
type -t grep

nightly-backup: not found in PATH
/usr/local/bin
/usr/bin
/bin
~/bin/nightly-backup
Nightly backup finished
cd is a shell builtin
file

To make a PATH change permanent, add the PATH=… line to ~/.bashrc or ~/.profile (many distributions already add ~/bin and ~/.local/bin when those directories exist at login). Other things to know:

• Not installed: Ubuntu suggests the package ("Command 'tree' not found, but can be installed with: sudo apt install tree"). Otherwise ask the package manager which package provides the file: dnf provides '*/semanage' on RHEL, apt-file search bin/semanage on Debian and Ubuntu (see Package Management).
• Only root has it: administrative programs live in /usr/sbin, which may not be in an ordinary user's PATH on some distributions; with sudo, the secure_path from sudoers is used instead.
• Moved program: bash remembers where it found a command. If a program moved (for example from /usr/local/bin to /usr/bin) and bash still looks in the old place, hash -r clears that memory.
• A script in the current directory: deploy.sh is not found, ./deploy.sh is. The current directory is deliberately not in PATH, so that a malicious file named ls in a shared directory cannot run in place of the real one.

A slow system and high load

"The server is slow" first needs a number. The load average from uptime or top is the average number of processes that are running, waiting for a CPU, or (on Linux) waiting for disk I/O, over 1, 5 and 15 minutes (see Logs and Monitoring). Compare it with the number of CPUs (nproc): a load of 8 is busy on a 2-CPU machine and relaxed on a 32-CPU one. Then find out what the load consists of. The CPU line in top tells you:

$ top

top - 10:14:03 up 12 days,  3:02,  2 users,  load average: 7.84, 6.91, 4.12
Tasks: 214 total,   1 running, 213 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.1 us,  2.0 sy,  0.0 ni,  9.6 id, 84.1 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :   3915.2 total,    182.4 free,   2604.9 used,   1398.0 buff/cache
MiB Swap:   2048.0 total,   1990.3 free,     57.7 used.   1310.3 avail Mem

Here wa is 84%: the load of almost 8 consists of processes waiting for the disk, not of work for the CPU. Such processes are in state D (uninterruptible sleep), and they are easy to list:

$ ps -eo state,pid,user,cmd | awk '$1 == "D"'
$ vmstat 1 3

D    2981 postgres postgres: autovacuum worker shop
D    3012 backup   tar -czf /mnt/nas/web1-2026-09-30.tar.gz /srv/data
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  6  59080 186752  40120 1391500    0    0 48210  1320  912 1450  4  2 10 84  0
 0  7  59080 184120  40120 1393880    0    0 51032   988  977 1502  3  2  9 86  0
 1  6  59080 183904  40124 1394212    0    0 49876  1104  941 1477  4  2 11 83  0

A backup running during office hours is reading the disk flat out. In vmstat, b counts processes blocked on I/O and bi is the amount read from disk; the columns si and so show swapping, which is zero here, so memory is not the problem. The fix is scheduling (see Scheduling Tasks) or running the job with lower I/O priority (ionice -c3).

When memory runs out

When a system runs out of memory and swap, the kernel's out-of-memory (OOM) killer ends a process to save the rest, usually the one using the most memory. To the user, a service simply died. The kernel log says so:

$ journalctl -k -b | grep -i 'out of memory'

Sep 30 11:02:47 web1 kernel: Out of memory: Killed process 2211 (java) total-vm:6123456kB, anon-rss:3012345kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:6400kB oom_score_adj:0

Restarting the service brings it back, but only for a while. Find out whether the program leaks memory, is configured to use too much (a Java heap size, a database cache), or the machine simply needs more memory. How memory and swap work is explained in Memory Management.

Boot problems

When a server does not come up, the question is how far it gets. Every stage of the boot has its own typical failures and its own way in:

The GRUB menu

GRUB (GRand Unified Bootloader) is the boot loader of most Linux distributions: the program that the firmware starts, which shows a menu of installed kernels, loads the chosen kernel and its initramfs into memory, and passes it the kernel parameters. Many servers skip the menu when there is only one system installed. To see it on Ubuntu, hold Shift (BIOS) or press Esc (UEFI) right after the firmware screen. In the menu you can:

• Choose Advanced options and boot the previous kernel, the quickest way out when a new kernel does not boot.
• Press e to edit the selected entry for this one boot: find the line that starts with linux, add a parameter at its end, and press Ctrl+X or F10 to boot. Nothing is saved; the next boot is normal again.

linux   /vmlinuz-6.8.0-85-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro systemd.unit=rescue.target

Permanent kernel parameters go in GRUB_CMDLINE_LINUX in /etc/default/grub, followed by sudo update-grub (Debian/Ubuntu); on RHEL, use sudo grubby --update-kernel=ALL --args='…'. On a cloud VM without a screen, the provider's serial console gives you the same menu.

Rescue and emergency mode

systemd offers two minimal states for repairs, chosen with a kernel parameter in GRUB, or entered automatically when the boot fails:

• Rescue mode (systemd.unit=rescue.target) is a single-user state in which all local file systems are mounted and only basic system services run, with a root shell on the console and no network logins. Use it when the system boots but something in the normal startup (a service, a network setting) needs fixing.
• Emergency mode (systemd.unit=emergency.target) is the most minimal state: only the root file system is mounted (read-only if you booted straight into it), and almost nothing else is started. systemd enters it by itself when a required file system cannot be mounted. Use it when even rescue mode fails.

Both ask for the root password. On Ubuntu, where root has no password, they answer Cannot open access to console, the root account is locked. You then need the init=/bin/bash route or a live USB stick.

Recovering from a broken fstab

A mistyped UUID or a disk that has been removed (without nofail) makes the boot wait about 90 seconds and then stop:

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-3f6c2a0e\x2d8d4b\x2d4c1e\x2d9a57\x2d2b1f0c9e7d24.device - /dev/disk/by-uuid/3f6c2a0e-8d4b-4c1e-9a57-2b1f0c9e7d24.
[DEPEND] Dependency failed for srv-data.mount - /srv/data.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.
Give root password for maintenance
(or press Control-D to continue):

Log in with the root password and repair the file. On Ubuntu, where root has no password, boot with init=/bin/bash (see below) or a live USB stick and edit /etc/fstab from there. The root file system may still be read-only, so remount it first:

# mount -o remount,rw /
# cp /etc/fstab /etc/fstab.broken
# nano /etc/fstab               # fix the line, or put a # in front of it
# systemctl daemon-reload
# mount -a                      # no errors this time?
# systemctl reboot

(Here # is the root prompt.) Use blkid to find the correct UUID. If the disk is gone for good, commenting the line out is enough to get the system up; add nofail to data disks from then on (see /etc/fstab).

Checking a file system with fsck

After a power failure or disk errors (see dmesg), a file system may need a check. fsck (file system check) runs the right checker for the type, e2fsck for ext4:

$ sudo umount /srv/data
$ sudo fsck -f /dev/sdb1

fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
data: 1893/1310720 files (0.4% non-contiguous), 668912/5242368 blocks

Resetting a forgotten root password

Anyone at the console who can edit GRUB parameters can get a root shell without a password. That is how administrators recover a forgotten root password, and it is why physical or console access equals full control: protect it with a GRUB password, disk encryption (LUKS, Linux Unified Key Setup, asks for its passphrase before any of this is possible) and locked server rooms or a protected cloud console. The exact steps differ per distribution and version; follow your distribution's documentation. In outline:

RHEL and its rebuilds: add rd.break to the linux line. The boot stops inside the initramfs, with the real root file system read-only under /sysroot:

switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot
sh-5.1# passwd root
sh-5.1# touch /.autorelabel
sh-5.1# exit
switch_root:/# exit

touch /.autorelabel matters: the new /etc/shadow was written without an SELinux label, and without a relabel nobody can log in. The relabel at the next boot takes a few minutes.

Debian and Ubuntu: add init=/bin/bash to the linux line. The kernel starts a root shell instead of systemd:

root@(none):/# mount -o remount,rw /
root@(none):/# passwd ana
root@(none):/# sync
root@(none):/# exec /sbin/init

On Debian, passwd root; on Ubuntu, the administrator's own account (root stays locked). exec /sbin/init continues the normal boot.

Kernel panic

A kernel panic is a fatal error from which the Linux kernel cannot safely recover, so it stops the whole system and prints a message and call trace on the console, like a Windows stop error. At boot, the classic one is:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

The kernel could not find its root file system, often because the initramfs for a newly installed kernel is missing or incomplete. Boot the previous kernel from the GRUB menu, then rebuild the initramfs: sudo update-initramfs -u -k all on Debian and Ubuntu, sudo dracut -f --regenerate-all on RHEL. Panics while the system is running more often point to hardware (memory, disk controller) or a third-party kernel module. The kdump service can save a crash dump of the kernel's memory for the vendor to analyze, like a Windows crash dump.

When to ask for help, and what to include

Escalate when the problem is outside your rights or knowledge, when the fix carries a risk you cannot judge (production data, a database, a kernel), or when a time limit agreed in the ticket system runs out. Escalating early with good information is professional, not a weakness. A good request lets the next person start where you stopped:

System:     web1 (Ubuntu 24.04.3 LTS, kernel 6.8.0-85-generic), production web server
Symptom:    nginx does not start since Sep 30 09:14 UTC; shop.example.com returns no response
Changed:    /etc/nginx/sites-enabled/shop modified Sep 29 16:45 (change 4211, new redirect rules)
Error:      [emerg] unexpected ";" in /etc/nginx/sites-enabled/shop:14   (journalctl -u nginx -b)
Tried:      nginx -t confirms the error; previous version of the file is in /root/shop.bak
Not tried:  restoring the backup - it would undo the redirects that marketing asked for
Needed:     decision: restore the old file, or someone who knows the new rules fixes line 14

Always include versions (cat /etc/os-release, uname -r, the program's own --version), the exact error text copied rather than retyped, the relevant log lines with their time stamps, what changed, and what you already tried. Remove passwords, keys and personal data from anything you paste into a ticket or a public forum.