Processes and Jobs

What a process is, how the kernel schedules it and how parent and child processes relate is explained on Processes and Threads. This page is about the commands: finding a process, reading what the tools tell you, stopping it properly, and working with several commands at once in one terminal.

Listing processes with ps

ps (process status) prints a snapshot of processes. Without options it shows only the processes of the current terminal, which is rarely what you want. Two option styles survive from Unix history and both are everywhere in documentation: BSD (Berkeley Software Distribution) style without a dash (ps aux) and System V/POSIX (Portable Operating System Interface) style with dashes (ps -ef). Both list every process on the system (illustrative, shortened):

$ ps aux

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.1  22412 13288 ?        Ss   Sep19   0:21 /sbin/init
root         912  0.0  0.0  12020  7936 ?        Ss   Sep19   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
mysql       1187  8.3  5.2 2451208 412332 ?      Ssl  Sep19 102:14 /usr/sbin/mysqld
www-data    2290  2.0  0.2 226488 18204 ?        S    09:12   0:41 php-fpm: pool www
ana         4433  0.0  0.0   8920  5504 pts/0    Ss   10:02   0:00 -bash
ana         4830  0.0  0.0  11200  4480 pts/0    R+   10:14   0:00 ps aux

ps -ef shows a slightly different set: UID PID PPID C STIME TTY TIME CMD, with the parent process ID (PPID) that ps aux lacks. When you know what you are looking for, choose the columns yourself with -o, or filter by name with pgrep:

$ ps -o pid,ppid,stat,cmd
$ pgrep -a nginx
$ ps aux --sort=-%mem | head -n 4

    PID    PPID STAT CMD
   4433    4432 Ss   -bash
   4821    4433 S    sleep 300
   4830    4433 R+   ps -o pid,ppid,stat,cmd
1502 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
1503 nginx: worker process
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysql       1187  8.3  5.2 2451208 412332 ?      Ssl  Sep19 102:14 /usr/sbin/mysqld
ana         5120 12.1  3.9 3375100 311020 ?      Sl   09:40   4:02 /usr/bin/node server.js
www-data    2290  2.0  0.2 226488 18204 ?        S    09:12   0:41 php-fpm: pool www

ps aux | grep nginx works too, but also finds the grep itself; pgrep avoids that and prints just the PIDs (-a adds the command line). ps -ef --forest or pstree draws the parent–child tree.

A look into /proc

ps and top get their information from /proc, a virtual file system in which the kernel presents every process as a directory named after its PID. Nothing there is stored on disk; each read asks the kernel. $$ is the PID of the current shell, so this reads the shell's own status (the tab after the colon is part of the file):

grep -E '^(Name|State):' /proc/$$/status

Name:	bash
State:	S (sleeping)

The shell is sleeping because it is waiting for grep to finish. Other useful entries per process: cmdline (the full command line), environ (its environment variables, readable only by its owner), fd/ (its open files) and cwd (its working directory). /proc/self always points to the process that is reading it.

Process states

The STAT column (S in top) shows the state as one letter, sometimes followed by modifiers:

Modifiers: s session leader (usually a shell or daemon), + in the foreground process group of its terminal, l multithreaded, < high priority, N low priority (niced). So Ssl is a sleeping, multithreaded session leader — a typical daemon.

Watching live: top and htop

top is installed everywhere and refreshes every few seconds. Its header summarizes the whole system (illustrative):

$ top

top - 10:14:03 up 12 days,  3:02,  2 users,  load average: 0.48, 0.61, 0.55
Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.2 us,  1.6 sy,  0.0 ni, 91.9 id,  0.2 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :   7812.4 total,    934.2 free,   2710.6 used,   4436.1 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   5101.8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1187 mysql     20   0 2451208 412332  36420 S   8.3   5.2 102:14.61 mysqld
   5120 ana       20   0 3375100 311020  48112 S   4.0   3.9   4:02.18 node
   2290 www-data  20   0  226488  18204  12032 S   2.0   0.2   0:41.07 php-fpm8.3

• load average: the average number of processes running or waiting to run (plus those in state D) over 1, 5 and 15 minutes. Compare it with the number of CPU cores; Logs and Monitoring explains how to read it.
• %Cpu(s): us user programs, sy kernel, ni niced programs, id idle, wa waiting for I/O, st stolen by the hypervisor on a virtual machine. High wa points at storage, not at the CPU.
• Memory: Linux uses spare RAM as disk cache (buff/cache), so low free is normal; avail Mem is what programs can still get.
• %CPU per process is a percentage of one core, so a multithreaded program on a 4-core machine can show up to 400%.
• PR and NI: kernel priority and niceness, see below.

htop (usually a separate package; on Red Hat Enterprise Linux (RHEL) 9 it comes from Extra Packages for Enterprise Linux (EPEL)) shows the same data with per-core bars, colors, mouse support and scrolling. Its function keys are labeled at the bottom: F3 search, F4 filter, F5 tree view, F6 choose the sort column, F9 send a signal, F10 quit.

Signals

A signal is a small asynchronous notification, identified by a number and a name, that the kernel delivers to a process: "terminate", "stop", "continue", "the terminal went away". A program can install a handler for most signals to react in its own way, or ignore them; for the rest the kernel applies a default action, usually ending the process. The ones you need:

The numbers 1, 2, 3, 9 and 15 are the same on every Linux system; the others shown are those of x86 and ARM Linux. kill -l lists them all. Refer to signals by name (-TERM, -HUP) in scripts and the question does not arise.

SIGTERM first, SIGKILL last

SIGTERM (signal 15) is the polite request to terminate, and the default signal of kill: the program can catch it, finish writing its files, remove lock files, close database connections and then exit. SIGKILL (signal 9) is handled by the kernel itself: the process is removed immediately and cannot catch, block or ignore it, so nothing gets cleaned up. That is why the order is always: SIGTERM, wait a few seconds, and SIGKILL only for a process that does not respond.

You can watch a script handle SIGTERM. trap installs a handler in bash, and timeout sends SIGTERM to the command when the time is up (--foreground makes it signal only the script, not the sleep running inside it; --preserve-status returns the script's own exit status):

cat <<'EOF' > worker.sh
trap 'echo "worker: got SIGTERM, saving state and exiting"; exit 0' TERM
echo "worker: started"
while true; do sleep 0.1; done
EOF
timeout --foreground --preserve-status 1 bash worker.sh
echo "exit status: $?"

worker: started
worker: got SIGTERM, saving state and exiting
exit status: 0

A process that is ended by a signal instead of exiting on its own gets exit status 128 plus the signal number: 130 for SIGINT, 137 for SIGKILL, 143 for SIGTERM. That is how you recognize it in scripts and logs:

timeout --preserve-status 1 sleep 5
echo "ended by SIGTERM: $?"
timeout --preserve-status -s INT 1 sleep 5
echo "ended by SIGINT: $?"

ended by SIGTERM: 143
ended by SIGINT: 130

Exit status 137 in a container or service log usually means something sent SIGKILL — very often the kernel's out-of-memory killer.

kill, pkill and killall

kill sends a signal to a PID (or to a job, kill %1); pkill and killall send it to processes selected by name. You can signal your own processes; other users' processes need sudo (illustrative):

$ pgrep -a rsync
4620 rsync -a /srv/data/ backup01:/backups/data/
$ kill 4620                 # SIGTERM
$ kill -0 4620 && echo "still running"
still running
$ kill -9 4620              # only after waiting: SIGKILL
$ sudo kill -HUP 1502       # ask nginx's master process to reload its configuration
$ pkill -u ana -x node      # SIGTERM to processes named exactly "node" owned by ana
$ killall -TERM php-fpm8.3  # SIGTERM to every process with this exact name

kill -0 sends no signal at all; it only checks whether the process exists and may be signaled. For a service managed by systemd, prefer systemctl stop or systemctl reload (see Services and systemd): systemd would otherwise restart what you just killed, or not know why it stopped.

Priority and niceness

Niceness is a per-process value from −20 to 19 that tells the Linux scheduler how much CPU time to give a process when several compete: a higher value is "nicer" to others and gets a smaller share, −20 gets the largest share, and the default is 0. It only matters when the CPU is busy; an idle system gives every process what it asks for. nice starts a command with a changed niceness (10 by default) and, on its own, prints the current value:

nice
nice -n 10 nice
nice -n 10 bash -c 'nice -n 5 nice'

0
10
15

A child inherits its parent's niceness, and changes add up. renice changes a running process. Ordinary users may only raise their processes' niceness (lower their priority); lowering it, including back to 0, requires root (illustrative):

$ nice -n 19 tar -czf /backup/home.tar.gz /home     # a backup that should not slow the server down
$ renice -n 5 -p 5120
5120 (process ID) old priority 0, new priority 5
$ renice -n 0 -p 5120
renice: failed to set priority for 5120 (process ID): Permission denied
$ sudo renice -n -5 -p 1187

Niceness affects CPU only. For a job that hammers the disk, ionice -c3 (idle I/O class) is the matching tool, but it only has an effect with the Budget Fair Queueing (BFQ) I/O scheduler.

Job control: foreground and background

Job control is the shell feature for running several commands from one terminal and moving them between foreground and background, stopping and resuming them. The shell numbers each pipeline you start as a job (%1, %2 …), separate from the PIDs of the processes in it.

Normally a command runs in the foreground: it owns the terminal's keyboard and the shell waits for it. A background process is one started from a shell that keeps running while the shell accepts new commands, because the command line ended with & or because it was moved there with bg; it cannot read from the keyboard.

A typical interactive moment: an archive job takes longer than expected and you need the terminal back (illustrative):

$ tar -czf projects.tar.gz ~/projects
^Z
[1]+  Stopped                 tar -czf projects.tar.gz ~/projects
$ bg
[1]+ tar -czf projects.tar.gz ~/projects &
$ sleep 600 &
[2] 5391
$ jobs
[1]-  Running                 tar -czf projects.tar.gz ~/projects &
[2]+  Running                 sleep 600 &
$ fg %1
tar -czf projects.tar.gz ~/projects
$
[2]+  Done                    sleep 600

The jobs list itself looks the same in a script. This runs two short background jobs, lists them, and waits for both:

sleep 1 &
sleep 2 &
jobs
wait
echo "both background jobs have finished"

[1]-  Running                 sleep 1 &
[2]+  Running                 sleep 2 &
both background jobs have finished

Scripts do not have full job control (no Ctrl+Z, no fg) and do not print the [1] 5391 or Done lines; those are for interactive shells. Jobs belong to the shell that started them: another terminal's jobs does not see them, but ps does.

Keeping commands running after you log out

When you close the terminal window or your SSH connection drops, the kernel sends SIGHUP ("hang up") to the session's shell, and bash passes SIGHUP on to all its jobs before it exits. Their default reaction is to terminate — which is why a long copy started with & over SSH dies with the connection. There are several ways around it:

• nohup (no hang-up) runs a command with SIGHUP ignored, so it survives the end of the session. If standard output is a terminal, it redirects the output to nohup.out, because the terminal will be gone. Add & to put it in the background as well.
• disown removes an already running job from the shell's job table, so the shell will not send it SIGHUP. Useful when you forgot nohup. It does not redirect output, so it works only if the job does not write to the terminal.
• A terminal multiplexer such as tmux keeps the whole session, including interactive programs, running on the server; you detach and later reattach, even from another computer. This is the best tool for interactive work over SSH; see SSH and Remote Work.
• For anything that must run permanently or start at boot, write a systemd service instead (Services and systemd), or schedule it (Scheduling Tasks).

$ nohup ./import-orders.sh &
[1] 6012
nohup: ignoring input and appending output to 'nohup.out'
$ ./rebuild-index.sh > rebuild.log 2>&1 &
[2] 6020
$ disown %2
$ jobs
[1]+  Running                 nohup ./import-orders.sh &
$ exit

After the logout, both scripts keep running and can be found with ps or pgrep, with PPID 1 (or that of a per-user systemd process), because their parent shell is gone. Give nohup'd and disowned commands their own log file, as the second line does; otherwise their output is lost or piles up in nohup.out.