Services and systemd

What systemd is

When the kernel has finished starting, it runs exactly one program and gives it process ID (PID) 1. That program, the init system, starts everything else: it mounts the file systems, brings up the network, starts the daemons that provide services, and finally the login prompt or the desktop. When you shut down, it stops them all again in the right order.

On almost every current distribution — Debian, Ubuntu, RHEL (Red Hat Enterprise Linux), Fedora, SUSE, Arch — that program is systemd: the init system and service manager that starts the system, starts services in parallel based on their dependencies, supervises them (and restarts them if configured to), and records their output in a central log, the journal. It replaced the older SysV (System V) init scripts in /etc/init.d, which started services one after another from shell scripts.

$ ps -p 1 -o pid,comm
$ systemctl --version

    PID COMMAND
      1 systemd
systemd 255 (255.4-1ubuntu8.10)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL …

Old habits still work: service nginx restart and /etc/init.d/nginx restart are passed on to systemd on most systems. Learn systemctl anyway; it is what the documentation, the exams and every error message refer to.

Units and unit types

systemd manages units: anything it can start, stop or watch. Each unit is described by a unit file, a small text file in the INI (initialization file) style (sections in square brackets, Key=value lines) whose name ends in the unit type. The file says what to run, what it depends on, and when it should start.

You will also meet .path (start a service when a file changes), .device and .slice units. In commands, the .service suffix may be left out: systemctl status nginx means nginx.service.

Where unit files live

Controlling services with systemctl

systemctl is the command-line tool for talking to systemd: starting, stopping, enabling and inspecting units and changing the system state. Looking is allowed for every user; changing needs root, so those commands run with sudo.

Reading systemctl status

$ systemctl status nginx

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-09-30 08:14:02 UTC; 2h 3min ago
       Docs: man:nginx(8)
    Process: 1021 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1024 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 1026 (nginx)
      Tasks: 3 (limit: 4556)
     Memory: 4.2M (peak: 6.1M)
        CPU: 38ms
     CGroup: /system.slice/nginx.service
             ├─1026 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─1027 "nginx: worker process"
             └─1028 "nginx: worker process"

Sep 30 08:14:02 web01 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Sep 30 08:14:02 web01 systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.

• The dot in front: a filled ● (often green) is running, a white ○ is inactive, a red × has failed.
• Loaded: which unit file systemd read, and whether the unit is enabled (starts at boot) or disabled. The preset is the distribution's default for new installs.
• Active: the state right now — active (running), inactive (dead) (not started, not an error), or failed — and since when.
• Main PID and CGroup: the main process and every process that belongs to the service. systemd tracks them with a control group (cgroup), so a stop really stops all of them.
• The last log lines of the unit, from the journal. For a failed service, the reason is usually here.

systemctl status opens in a pager when the output is long; press q to leave, or add --no-pager. A failed service looks like this (captured before Restart=on-failure was added to the unit file; with it, systemd keeps retrying and the status shows activating (auto-restart) between attempts):

$ systemctl status inventory-api

× inventory-api.service - Inventory API (internal web application)
     Loaded: loaded (/etc/systemd/system/inventory-api.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2026-09-30 10:02:17 UTC; 4min ago
   Duration: 12ms
    Process: 2311 ExecStart=/opt/inventory/venv/bin/python -m inventory.server --port 8080 (code=exited, status=203/EXEC)
   Main PID: 2311 (code=exited, status=203/EXEC)
        CPU: 4ms

Sep 30 10:02:17 web01 systemd[1]: Started inventory-api.service - Inventory API (internal web application).
Sep 30 10:02:17 web01 (python)[2311]: inventory-api.service: Failed to locate executable /opt/inventory/venv/bin/python: No such file or directory
Sep 30 10:02:17 web01 (python)[2311]: inventory-api.service: Failed at step EXEC spawning /opt/inventory/venv/bin/python: No such file or directory
Sep 30 10:02:17 web01 systemd[1]: inventory-api.service: Main process exited, code=exited, status=203/EXEC
Sep 30 10:02:17 web01 systemd[1]: inventory-api.service: Failed with result 'exit-code'.

status=203/EXEC is systemd's own code for "the program in ExecStart could not be executed" — here because the path is wrong. Statuses from 200 up, shown with a name such as 203/EXEC, 200/CHDIR or 217/USER, are systemd's own: the program never started. Other exit statuses come from the program itself, and its own error message is then in the log lines.

start, stop, restart and reload

These commands print nothing when they succeed. Check the result with systemctl status or the short checks is-active, is-enabled and is-failed, which print one word and set the exit status, so scripts can use them:

$ sudo nginx -t && sudo systemctl reload nginx
$ systemctl is-active nginx
$ systemctl is-enabled nginx

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
active
enabled

Finding failed units

$ systemctl list-units --failed
$ systemctl list-units --type=service --state=running

  UNIT                  LOAD   ACTIVE SUB    DESCRIPTION
● inventory-api.service loaded failed failed Inventory API (internal web application)

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

1 loaded units listed.
…

list-units --failed is a good first command on any server you log in to after a problem report. After fixing the cause, sudo systemctl reset-failed inventory-api clears the failed state. systemctl list-unit-files --type=service lists every installed service with its enabled or disabled state, including those that are not running.

Starting versus enabling

This is the most important distinction on this page. start acts now and is forgotten at the next reboot. enable acts at boot: an enabled service is a unit that systemd starts automatically when the system boots, because enable linked it into a target's list of wanted units. Enabling does not start it now, and starting does not enable it.

$ sudo systemctl enable nginx
$ sudo systemctl enable --now nginx
$ sudo systemctl disable --now nginx

Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service.
Removed "/etc/systemd/system/multi-user.target.wants/nginx.service".

The output shows what enabling really is: a symbolic link in /etc/systemd/system/multi-user.target.wants/, created from the [Install] section of the unit file (see below). The second command printed nothing because the link already existed; --now also started the service. disable --now removes the link and stops it. On Debian and Ubuntu, nginx is already enabled right after installation, and each enable or disable there also prints two lines about systemd-sysv-install (the package still ships an old init script, which is kept in sync).

mask links the unit to /dev/null; use it for a service that must never run (for example one of two conflicting firewall services), and unmask to undo it.

Targets

A target is a unit that groups other units and represents a state of the system, such as "fully booted with networking, no desktop". Booting means reaching the default target, and everything that target wants — directly or through other targets — is started.

$ systemctl get-default
$ sudo systemctl set-default multi-user.target

graphical.target
Removed "/etc/systemd/system/default.target".
Created symlink /etc/systemd/system/default.target → /usr/lib/systemd/system/multi-user.target.

set-default changes the target for the next boots; sudo systemctl isolate multi-user.target switches right away (and stops everything the new target does not want, such as the desktop). Rescue and emergency mode are recovery tools, usually chosen at boot time; when and how to use them is covered in Troubleshooting Linux.

A unit file, line by line

Suppose a small internal web application should run permanently on a server, restart if it crashes, and start at boot. Its unit file, written by you, belongs in /etc/systemd/system/:

[Unit]
Description=Inventory API (internal web application)
Documentation=https://wiki.example.com/inventory-api
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=inventory
Group=inventory
WorkingDirectory=/opt/inventory
ExecStart=/opt/inventory/venv/bin/python -m inventory.server --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Then tell systemd about the new file, and enable and start it in one go:

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now inventory-api
$ systemctl status inventory-api

daemon-reload makes systemd re-read all unit files. It is needed every time you create or change a unit file by hand; without it systemd keeps using the old version (and status warns: Warning: The unit file, source configuration file or drop-ins of inventory-api.service changed on disk. Run 'systemctl daemon-reload' to reload units.). Note the difference: daemon-reload reloads systemd's unit files, while reload nginx makes nginx re-read its own configuration.

Changing a service: overrides with systemctl edit

Vendor unit files in /usr/lib/systemd/system/ belong to the package: the next update overwrites your changes without warning. To change a setting, add a drop-in: a small file with only the lines you want to change, which systemd reads after the original.

$ sudo systemctl edit nginx

This opens an editor on /etc/systemd/system/nginx.service.d/override.conf (created if needed), shows the original file as comments for reference, and runs daemon-reload for you when you save. To make nginx restart automatically after a crash, the override only needs:

[Service]
Restart=on-failure
RestartSec=10

$ sudo systemctl restart nginx
$ systemctl cat nginx

# /usr/lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
…

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
Restart=on-failure
RestartSec=10

systemctl cat shows the complete, effective configuration with the file each part comes from — the first thing to check when a service does not behave as its unit file suggests. sudo systemctl revert nginx removes all your drop-ins and returns to the vendor version. systemctl edit --full copies the whole unit to /etc/systemd/system/ instead; that works, but then you no longer receive the vendor's fixes to it.

Reading a service's log

Everything a service writes to its standard output and standard error ends up in the systemd journal, together with systemd's own messages about that unit. journalctl reads it; these options cover most service problems:

$ journalctl -u nginx --since "1 hour ago"
$ journalctl -u inventory-api -b
$ journalctl -u inventory-api -f
$ journalctl -u inventory-api -n 50 --no-pager

A useful habit while testing: keep journalctl -u name -f running in a second terminal while you start or restart the service in the first. Reading logs in depth — priorities, the kernel log, classic log files and log rotation — is covered in Logs and Monitoring.