Linux Security Basics

The mindset: least privilege, layers, less

Security on a Linux server is not one product you install. It is a set of habits built on three ideas, the same ones behind network security (see Network Security Basics):

• Least privilege: every account, service and program gets only the rights it needs. A web server that runs as its own unprivileged user can, when it is broken into, do far less than one that runs as root.
• Defense in depth: several independent layers, so that one mistake is not the end. A firewall, key-only SSH, updates, file permissions and SELinux each stop attacks the others miss.
• A small attack surface: every running service, open port, installed package and user account is something that can be attacked. What is not there cannot be exploited.

Hardening is the process of reducing a system's attack surface and tightening its configuration beyond the defaults: removing software and accounts that are not needed, closing ports, restricting services and enabling security features. Organizations usually follow a published baseline, such as the CIS (Center for Internet Security) Benchmarks or the DISA (Defense Information Systems Agency) STIGs (Security Technical Implementation Guides), rather than inventing their own list. A first pass on any server:

$ systemctl list-units --type=service --state=running   # what runs?
$ sudo ss -tlnp                                          # what listens on the network?
$ apt list --installed | wc -l                           # how much is installed? (rpm -qa | wc -l on RHEL)

For every service and listening port, you should be able to say why it is there. If nobody knows, find out, then disable and remove it (see Services and systemd and Package Management).

Updates, automatically

Most successful attacks use vulnerabilities for which a fix has already been published. Installing security updates quickly is therefore the single most effective measure, and the easiest way to make it happen is to let the system do it. Unattended upgrades are updates that a system downloads and installs by itself on a schedule, typically limited to security fixes: the unattended-upgrades package on Debian and Ubuntu, dnf-automatic on RHEL and Fedora.

Fedora's newer dnf5 packages the same feature slightly differently (dnf5-automatic.timer); check your release's documentation. A dry run shows what would be installed without changing anything:

$ sudo unattended-upgrade --dry-run --debug

Two things are easy to forget. First, an updated library or kernel is only used after the programs using it restart, and a new kernel only after a reboot; plan regular maintenance reboots, or enable the automatic reboot at a quiet hour (Unattended-Upgrade::Automatic-Reboot "true";). Second, a release that no longer receives updates cannot be kept safe at all: know the end-of-support date of every system you run.

Users and sudo

Accounts, groups and sudo rules are explained in Users, Groups and Permissions in Practice. For security, check these regularly:

$ getent group sudo wheel              # who may become root? (sudo on Debian/Ubuntu, wheel on RHEL)
$ sudo -l                              # what may I run with sudo?
$ awk -F: '$3 == 0' /etc/passwd        # accounts with user ID 0 (should be only root)
$ sudo journalctl _COMM=sudo --since today   # what was run with sudo today?

sudo:x:27:ana,marco
root:x:0:0:root:/root:/bin/bash

• Everyone logs in with their own account, so the logs show who did what. No shared admin accounts, no root logins.
• Remove people from sudo/wheel and lock their accounts when they leave or change roles (sudo usermod -L -e 1 marco locks the password and expires the account).
• Service accounts get no login shell (/usr/sbin/nologin) and no sudo rights.
• Give narrow sudo rules where possible ("may restart nginx") instead of ALL.

Remote access: SSH, fail2ban and the firewall

A server on the internet receives login attempts within minutes of being switched on. The SSH hardening steps themselves (keys instead of passwords, no root login, a limited set of users) are described in Remote Access and Management, and daily work with keys in SSH and Remote Work. Two layers complete the picture.

fail2ban

fail2ban is a service that watches log files or the journal for repeated failed logins and temporarily blocks the offending IP addresses in the firewall. Each protected service is a jail with its own rules. Settings go in /etc/fail2ban/jail.local, never in jail.conf, which package updates overwrite:

[DEFAULT]
# never ban the office network or the jump host
ignoreip = 127.0.0.1/8 ::1 192.0.2.0/24

[sshd]
enabled  = true
maxretry = 5
findtime = 10m
bantime  = 1h

Five failures within ten minutes earn a one-hour ban. After sudo systemctl restart fail2ban, check the jail:

$ sudo fail2ban-client status sshd

Status for the jail: sshd
|- Filter
|  |- Currently failed: 2
|  |- Total failed:     37
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 1
   |- Total banned:     4
   `- Banned IP list:   203.0.113.45

sudo fail2ban-client set sshd unbanip 203.0.113.45 lifts a ban, for example when a colleague locked themselves out. If the system has no /var/log/auth.log (Debian 12 logs only to the journal by default), add backend = systemd to the jail.

The host firewall

A host firewall on the server itself allows only the ports that are meant to be open, even if a service is accidentally started or listens on every interface. The rule is default deny for incoming traffic plus explicit exceptions. On Ubuntu with ufw:

$ sudo ufw default deny incoming
$ sudo ufw allow OpenSSH          # before enabling, or you lock yourself out
$ sudo ufw allow 443/tcp
$ sudo ufw enable

On RHEL, firewalld is active by default (sudo firewall-cmd --list-all). Both are explained in Networking from the Command Line.

Mandatory access control: SELinux and AppArmor

Ordinary Linux permissions are discretionary: the owner of a file decides who may use it, and a process may do everything its user may do. If an attacker takes over the web server process, they can read every file the web server's user can read. Mandatory access control (MAC) adds a second check that is set by a system-wide policy and that even the file owner cannot override: the web server may only touch web content, whatever the file permissions say. Both checks must say yes.

The Red Hat family uses SELinux; Debian, Ubuntu and their relatives use AppArmor. You will normally meet only the one of your distribution.

SELinux

SELinux (Security-Enhanced Linux) is a mandatory access control system in the Linux kernel that labels every process, file and port with a security context and allows an action only if the loaded policy has a rule for that combination of labels. It is enabled and enforcing by default on RHEL, Fedora, Rocky Linux and AlmaLinux. It has three modes:

$ getenforce
$ ls -Z /var/www/html/
$ ps -eZ | grep httpd

Enforcing
unconfined_u:object_r:httpd_sys_content_t:s0 index.html
unconfined_u:object_r:user_home_t:s0 prices.html
system_u:system_r:httpd_t:s0       1311 ?        00:00:00 httpd

A context has four parts, user:role:type:level, and in the default ("targeted") policy the type is what counts. The web server runs as type httpd_t, and the policy lets httpd_t read files of type httpd_sys_content_t. index.html has that type; prices.html has user_home_t, because someone created it in their home directory and then moved it with mv, which keeps the label. The web server gets "403 Forbidden" for that file, even though its permissions are fine. The denial is in the audit log, as an AVC (access vector cache) message:

$ sudo ausearch -m AVC -ts recent

type=AVC msg=audit(1790758800.412:318): avc:  denied  { read } for  pid=1311 comm="httpd" name="prices.html" dev="dm-0" ino=1049123 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

Read it as: denied read by httpd (source type httpd_t) on a file of target type user_home_t. The fix is to give the file the label its location should have. restorecon resets labels to the policy's defaults for that path:

$ sudo restorecon -Rv /var/www/html

Relabeled /var/www/html/prices.html from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:httpd_sys_content_t:s0

Other everyday fixes, each targeted at one cause:

• Content in a non-standard directory: record the right default label for the path, then apply it: sudo semanage fcontext -a -t httpd_sys_content_t "/srv/web(/.*)?" followed by sudo restorecon -Rv /srv/web.
• A feature switched off by a boolean: the policy has on/off switches for common needs. getsebool -a | grep httpd lists them; sudo setsebool -P httpd_can_network_connect on lets the web server open network connections (for a reverse proxy). -P makes it permanent.
• A service on a non-standard port: sudo semanage port -a -t http_port_t -p tcp 8081 (if semanage port -l | grep 8081 shows the port already has another type, use -m instead of -a).

sudo setenforce 0 switches to permissive mode until the next reboot, and setenforce 1 back. It is a diagnostic tool: if the problem disappears in permissive mode, SELinux is involved, and the audit log says why. The permanent mode is set in /etc/selinux/config.

AppArmor

AppArmor is a mandatory access control system in the Linux kernel that confines individual programs with per-program profiles listing the files, capabilities and network access each may use. It works with file paths rather than labels, which makes profiles easier to read. It is enabled by default on Ubuntu and on Debian since version 10. Programs without a profile are not restricted by it. Each profile is in one of two modes: enforce (violations blocked and logged) or complain (only logged, like SELinux's permissive mode, but per program).

$ sudo aa-status

apparmor module is loaded.
112 profiles are loaded.
38 profiles are in enforce mode.
   /usr/bin/man
   /usr/sbin/chronyd
   /usr/sbin/cupsd
   ...
4 profiles are in complain mode.
   ...
70 profiles are in unconfined mode.
3 processes have profiles defined.
3 processes are in enforce mode.
   /usr/sbin/chronyd (812)
   /usr/sbin/cupsd (1022)
   /usr/sbin/cups-browsed (1047)
0 processes are in complain mode.

Profiles live in /etc/apparmor.d/. Denials appear in the kernel log as lines containing apparmor="DENIED" (sudo journalctl -k | grep DENIED), with the profile and the path that was refused. The tools aa-complain and aa-enforce (package apparmor-utils) switch a profile's mode; as with SELinux, complain mode is for diagnosing, and the fix is to adjust the profile (often a local addition in /etc/apparmor.d/local/), not to leave it off.

Auditing file permissions

Two kinds of files deserve a regular look. Setuid and setgid programs run with the rights of their owner (often root) instead of the user who starts them, so an unexpected one is a possible back door. World-writable files and directories can be changed by every account, so a script or configuration file among them lets any user change what another one runs. find -perm finds both; here on a small tree with deliberate problems:

umask 022
mkdir -p srv/app/bin srv/app/uploads srv/app/config srv/shared
printf '#!/bin/sh\necho report\n' > srv/app/bin/report
printf 'db_host=localhost\n' > srv/app/config/app.conf
chmod 4755 srv/app/bin/report          # setuid: runs as the file's owner
chmod 666 srv/app/config/app.conf      # anyone may change the configuration
chmod 777 srv/app/uploads              # world-writable, no sticky bit
chmod 1777 srv/shared                  # world-writable, with sticky bit (like /tmp)

echo "== setuid or setgid files"
find . -type f \( -perm -4000 -o -perm -2000 \) | sort
echo "== world-writable files"
find . -type f -perm -o+w | sort
echo "== world-writable directories without sticky bit"
find . -type d -perm -o+w ! -perm -1000 | sort

== setuid or setgid files
./srv/app/bin/report
== world-writable files
./srv/app/config/app.conf
== world-writable directories without sticky bit
./srv/app/uploads

-perm -4000 means "at least these bits set". A world-writable directory with the sticky bit, like /tmp or srv/shared here, is normal: users can only delete their own files in it. On a real system, search from / and stay on local disks:

$ sudo find / -xdev -type f -perm -4000 2>/dev/null | sort

/usr/bin/chfn
/usr/bin/chsh
/usr/bin/gpasswd
/usr/bin/mount
/usr/bin/newgrp
/usr/bin/passwd
/usr/bin/su
/usr/bin/sudo
/usr/bin/umount
/usr/lib/openssh/ssh-keysign

These are expected: they come from packages and need root rights to do their job (passwd writes /etc/shadow). Save the list and compare it after each audit; a setuid file in /home, /tmp or /var/www is a red flag. Fix world-writable files with chmod o-w, never with chmod 777 on something else.

Watching the logs

Authentication events (logins, sudo, SSH) are logged in /var/log/auth.log on Ubuntu (and on Debian when rsyslog is installed) and /var/log/secure on RHEL, and always in the journal (see Logs and Monitoring):

$ sudo journalctl _COMM=sshd --since "1 hour ago"
$ last -n 3                 # recent logins and reboots
$ sudo lastb -n 3           # recent failed logins

Sep 30 08:41:17 web1 sshd[3104]: Accepted publickey for ana from 192.0.2.25 port 51522 ssh2: ED25519 SHA256:q3Vd8KmC0rT7yPzW1sNf4hBxJ2eLuA9oGiR6cYkM5tE
Sep 30 08:41:17 web1 sshd[3104]: pam_unix(sshd:session): session opened for user ana(uid=1000) by (uid=0)
Sep 30 08:57:30 web1 sshd[3190]: Failed password for invalid user test from 198.51.100.7 port 40310 ssh2
ana      pts/0        192.0.2.25       Wed Sep 30 08:41   still logged in
ana      pts/0        192.0.2.25       Tue Sep 29 16:02 - 17:30  (01:27)
reboot   system boot  6.8.0-85-generic Tue Sep 29 07:58   still running
test     ssh:notty    198.51.100.7     Wed Sep 30 08:57 - 08:57  (00:00)
root     ssh:notty    203.0.113.45     Wed Sep 30 03:13 - 03:13  (00:00)
admin    ssh:notty    198.51.100.7     Wed Sep 30 03:13 - 03:13  (00:00)

A stream of failures from strangers is normal on the internet and is exactly what fail2ban handles. What deserves attention is a successful login you cannot explain: an unusual time, an unknown address, an account that should not log in at all. On some newer releases, last is provided by wtmpdb and works the same way. Counting failures per address is a small text job:

cat > auth.log <<'EOF'
Sep 30 03:12:44 web1 sshd[2211]: Failed password for root from 203.0.113.45 port 52144 ssh2
Sep 30 03:12:47 web1 sshd[2211]: Failed password for root from 203.0.113.45 port 52144 ssh2
Sep 30 03:13:02 web1 sshd[2230]: Failed password for invalid user admin from 198.51.100.7 port 40022 ssh2
Sep 30 03:13:05 web1 sshd[2230]: Failed password for invalid user admin from 198.51.100.7 port 40022 ssh2
Sep 30 03:13:09 web1 sshd[2235]: Failed password for root from 203.0.113.45 port 52190 ssh2
Sep 30 03:13:12 web1 sshd[2235]: Failed password for root from 203.0.113.45 port 52190 ssh2
Sep 30 08:41:17 web1 sshd[3104]: Accepted publickey for ana from 192.0.2.25 port 51522 ssh2: ED25519 SHA256:q3Vd8KmC0rT7yPzW1sNf4hBxJ2eLuA9oGiR6cYkM5tE
Sep 30 08:57:30 web1 sshd[3190]: Failed password for invalid user test from 198.51.100.7 port 40310 ssh2
EOF

echo "Failed attempts per source address:"
grep 'Failed password' auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn
echo "Successful logins:"
grep 'Accepted' auth.log | awk '{print $1, $2, $3, $9, "from", $11}'

Failed attempts per source address:
      4 203.0.113.45
      3 198.51.100.7
Successful logins:
Sep 30 08:41:17 ana from 192.0.2.25

$(NF-3) is the fourth field from the end: the address always sits four words before the end of a Failed password line, whether or not the words "invalid user" are in it.

Checking installed files

The package manager knows a checksum for every file it installed, so it can tell you which ones were changed afterward:

$ sudo rpm -V openssh-server        # RHEL: one package (rpm -Va: all of them)
$ sudo dpkg --verify openssh-server # Debian/Ubuntu (or: sudo debsums -s)

S.5....T.  c /etc/ssh/sshd_config

Each position is one test that failed: S size, 5 checksum, T modification time; the c marks a configuration file. Changed configuration files are normal; you changed them. A changed program in /usr/bin or /usr/sbin is not. Keep in mind that an attacker with root rights can also change the package database; tools such as AIDE (Advanced Intrusion Detection Environment) compare against a baseline stored somewhere safe.

Safe habits: downloads and secrets

Never pipe the internet into a shell

Installation instructions sometimes say curl -fsSL https://example.com/install.sh | sudo bash. That runs code you have never seen, as root, straight from a web server. If the server is compromised, the domain has expired and been taken over, or the download breaks off halfway through a line like rm -rf /opt/tool, the result runs anyway. Do this instead: prefer the distribution's packages or the vendor's signed repository; otherwise download the script, read it, and verify it against a checksum or signature that the vendor publishes separately.

printf '#!/bin/sh\necho "Installing monitoring agent 2.4"\n' > install.sh
sha256sum install.sh > install.sh.sha256     # the checksum the vendor publishes
cat install.sh.sha256
sha256sum -c install.sh.sha256

# Someone changes the script after the checksum was published
printf 'echo "one extra line from an attacker"\n' >> install.sh
sha256sum -c install.sh.sha256

1021c0a80ddfba3187eb8252b406ef459c7975162635aa0ea3ad1acb2c4172ec  install.sh
install.sh: OK
install.sh: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

A checksum only helps if it comes from somewhere the attacker cannot also change; a checksum on the same compromised web page proves nothing. Signatures made with GPG (GNU Privacy Guard) are stronger, because the signing key is not on the web server. This is what package managers do for you automatically.

Secrets in scripts and environment variables

Passwords, API tokens and keys end up in places they should not be:

• In a script: everyone who can read the script, its backups or the Git repository it is in can read the secret.
• On the command line (mysqldump -pS3cret): every user on the system can see the arguments of every process with ps, and it lands in your shell history.
• In environment variables: better than the command line, since only the same user and root can read a process's environment, but every child process inherits them, and they tend to show up in debug output and crash reports.

The usual fix: store the secret in a file that only the service account can read (mode 600, created with a strict umask), let the program read it from there, and keep that file out of version control. For systemd services, LoadCredential= passes such a file to one service only. A quick search for hard-coded secrets, and a file created the safe way:

mkdir scripts
cat > scripts/backup.sh <<'EOF'
#!/bin/bash
API_TOKEN="tk_live_4f9a1c77e2"
mysqldump -u backup -pS3cret-2026 shop > shop.sql
EOF
cat > scripts/report.sh <<'EOF'
#!/bin/bash
# reads DB_PASSWORD from a file only the report account can read
source /etc/report/credentials
EOF

echo "== possible secrets in scripts"
grep -nE '(PASSWORD|TOKEN|SECRET)=|-p[^ ]' scripts/*.sh

echo "== a credentials file readable only by its owner"
umask 077
printf 'DB_PASSWORD=Tr0ub4dor-2026\n' > credentials
stat -c '%A %n' credentials

== possible secrets in scripts
scripts/backup.sh:2:API_TOKEN="tk_live_4f9a1c77e2"
scripts/backup.sh:3:mysqldump -u backup -pS3cret-2026 shop > shop.sql
== a credentials file readable only by its owner
-rw------- credentials

Secrets for scripts and for automation tools: Keeping secrets out of scripts and Secrets in Automation.