Kernel, System Calls and Drivers

Why the kernel needs protection

A computer runs dozens of programs from different sources at once: a browser running code from web pages, an email client, an update agent, a game. If any of them could write anywhere in memory or talk to the disk controller directly, one bug — or one piece of malware — could corrupt every other program's data, read passwords out of another program's memory, or wipe the disk.

So the operating system's kernel reserves direct hardware access for itself, and every other program has to ask. Software alone cannot enforce that: a program could simply ignore the rules. The enforcement comes from the CPU itself.

User mode and kernel mode

Modern CPUs can run code at different privilege levels. The operating system uses two of them:

• Kernel mode (also privileged mode or supervisor mode) is where the kernel and kernel-mode drivers run. Code in kernel mode can execute every instruction, access all memory and talk directly to devices.
• User mode is where applications — and shells, services and the desktop — run. Code in user mode can only reach its own memory, and the CPU refuses privileged instructions such as disabling interrupts, changing memory mappings or accessing device ports. Anything like that has to go through the kernel.

Privilege rings

On x86 processors (Intel and AMD) the privilege levels are called rings, numbered 0 to 3, with ring 0 the most privileged. Windows, Linux and macOS use only two: ring 0 for kernel mode and ring 3 for user mode. ARM processors, used in phones, Apple silicon Macs and many laptops, call them exception levels: EL0 for user programs, EL1 for the kernel, and higher levels for a hypervisor and secure firmware. Virtualization adds a level below the kernel for the hypervisor, informally called "ring −1"; see the Cloud topic.

When code crashes

The split is what makes crashes survivable. When a program in user mode does something illegal — reads memory that is not its own, divides by zero without handling it — the CPU stops it and hands control to the kernel. The kernel ends that one process and cleans up its memory and open files. Windows shows "… has stopped working" or the app simply closes and logs an Application Error event (Event ID 1000) in Event Viewer; Linux prints Segmentation fault. Every other program keeps running.

In kernel mode there is nobody above to clean up. If the kernel or a driver makes the same kind of mistake, the kernel cannot trust its own data anymore and deliberately stops the machine to prevent damage — a stop error (blue screen) on Windows, a kernel panic on Linux and macOS. That is why stop errors so often name a driver file (.sys) under "What failed", and why drivers deserve extra care (see below).

System calls

A system call (syscall) is a request from a program to the kernel to do something the program may not do itself: open or read a file, allocate memory, start a process, send network data, read the clock. It is the only way a program can deliberately ask the kernel to do something. It works like this:

1. The program puts a system call number (which service it wants) and its arguments in agreed CPU registers.
2. It executes a special instruction — syscall on x64, svc on ARM — that switches the CPU to kernel mode and jumps to one fixed entry point in the kernel. The program cannot choose where to jump, so it cannot sneak into the middle of kernel code.
3. The kernel looks up the number in its system call table, checks the arguments and the caller's permissions, and does the work.
4. It puts the result (or an error code) in a register and switches back to user mode, right after the instruction that made the call.

The permission check in step 3 is where "Access is denied" and "Permission denied" come from: the kernel refused the system call.

Examples on Linux and Windows

Programs almost never make system calls by hand. They call a library function that does it for them: on Linux and macOS the C library (open(), read()), on Windows the Windows API in kernel32.dll, which calls the native functions in ntdll.dll, which make the actual system call. Microsoft does not document the native calls and changes their numbers between Windows builds; the Windows API is the stable interface. That layering is why a program written for Windows 7 can still run on Windows 11.

On Linux you can watch the system calls a program makes with strace. Here, reading a small file with cat (shortened and illustrative: the real trace also shows the program loading its libraries, and the exact calls vary by version):

strace -e trace=openat,read,write,close cat notes.txt > /dev/null

…
openat(AT_FDCWD, "notes.txt", O_RDONLY) = 3
read(3, "Buy milk\n", 131072)           = 9
write(1, "Buy milk\n", 9)               = 9
read(3, "", 131072)                     = 0
close(3)                                = 0
…
+++ exited with 0 +++

Each line is one trip into the kernel: open the file (the kernel hands back file descriptor 3), read 9 bytes, write them to standard output (descriptor 1), read again and get 0 bytes (end of file), close. On Windows, Process Monitor from Microsoft's Sysinternals suite shows the same kind of detail for file and registry access, and is a favorite tool for "access denied" and "file not found" problems.

A toy system call table

The kernel side of a system call is, at heart, a table lookup. This simulation uses the real Linux x86-64 numbers for read (0), open (2) and close (3), and returns negative error numbers the way the Linux kernel does (-2 is ENOENT, "no such file", -9 is EBADF, "bad file descriptor", -38 is ENOSYS, "function not implemented"):

# A toy system call table: number -> kernel function
files = {"notes.txt": "Buy milk\n"}
open_files = {}

def sys_open(path):
    if path not in files:
        return -2                     # ENOENT: no such file
    fd = 3 + len(open_files)          # 0, 1, 2 are stdin, stdout, stderr
    open_files[fd] = path
    return fd

def sys_read(fd):
    return files[open_files[fd]] if fd in open_files else -9   # EBADF

def sys_close(fd):
    return 0 if open_files.pop(fd, None) else -9

SYSCALL_TABLE = {0: sys_read, 2: sys_open, 3: sys_close}

def syscall(number, arg):
    """What the trap does: switch to kernel mode, look up, run, return."""
    handler = SYSCALL_TABLE.get(number)
    result = handler(arg) if handler else -38          # ENOSYS
    print(f"syscall {number} ({handler.__name__ if handler else '?'}) -> {result!r}")
    return result

fd = syscall(2, "notes.txt")
syscall(0, fd)
syscall(3, fd)
syscall(0, fd)               # reading a closed file fails
syscall(2, "secret.txt")     # file does not exist
syscall(99, None)            # no such system call

syscall 2 (sys_open) -> 3
syscall 0 (sys_read) -> 'Buy milk\n'
syscall 3 (sys_close) -> 0
syscall 0 (sys_read) -> -9
syscall 2 (sys_open) -> -2
syscall 99 (?) -> -38

The point to notice: the program only ever supplies a number and an argument. The kernel decides what that number means and checks everything before acting, so a bad request gets an error code, not access to someone else's data.

Interrupts

System calls are how programs get the kernel's attention. Hardware needs a way too: a key was pressed, a network packet arrived, the SSD finished reading a block. An interrupt is a signal that makes the CPU stop what it is doing, switch to kernel mode and run a small kernel routine — an interrupt handler (interrupt service routine, ISR) — for that event, then continue where it left off.

• Hardware interrupts come from devices, through an interrupt request line (IRQ): keyboard, mouse, network adapter, storage controller, and the timer on the motherboard or in the CPU. Without interrupts the CPU would have to keep asking every device "anything new?" (polling), wasting time.
• Software interrupts and exceptions are raised by the CPU itself while running an instruction: a division by zero, an access to memory that is not mapped (a page fault; see Memory Management), or, on older systems, a deliberate trap instruction used to make a system call.

The timer interrupt deserves special attention. The kernel programs a hardware timer to interrupt the CPU many times per second. Each time, the kernel gets control back — even from a program stuck in an endless loop — and can decide to give the CPU to someone else. That is the foundation of preemptive multitasking, explained in CPU Scheduling.

Kernel designs

Every kernel must provide scheduling, memory management, file systems, networking and drivers. Designs differ in how much of that runs in kernel mode.

Monolithic kernels

A monolithic kernel runs all core services — scheduler, memory manager, file systems, network stack and device drivers — together in kernel mode, in one shared address space. Components call each other directly, which is fast and simple. The cost is that a bug in any part, even an obscure driver, can crash or compromise the whole kernel. Linux and the BSD family are monolithic. Linux softens the "one big program" aspect with loadable modules (below), but a loaded module still runs in kernel mode with full rights.

Microkernels

A microkernel keeps only the bare minimum in kernel mode — typically scheduling, basic memory management and message passing between processes — and runs file systems, network stacks and drivers as separate user-mode programs (servers). A crashing driver is then just a crashing process that can be restarted. The price is speed: every request between components is a message that crosses the user/kernel boundary. Microkernels are popular where reliability matters most: QNX runs in cars and industrial systems, seL4 is known for a mathematical proof that its code matches its specification, and MINIX 3 was designed for teaching and self-repair.

Hybrid kernels

Windows and macOS are usually described as hybrid kernels. The Windows NT kernel has a layered, microkernel-inspired structure, and macOS's XNU combines the Mach microkernel with parts of BSD Unix. In both, however, the file systems, network stack and most drivers run in kernel mode for speed — much like a monolithic kernel. Windows even moved most of its window and graphics system into kernel mode in Windows NT 4.0. "Hybrid" is best read as "microkernel ideas in the structure, monolithic in practice".

Kernel modules and drivers

A device driver is the code that knows how to talk to one kind of hardware. Because it must touch the device directly, most drivers run in kernel mode, as part of the kernel.

Nobody wants to rebuild the kernel for every new printer, so modern kernels can load code while they run. A kernel module is a piece of kernel code — usually a driver or a file system — that is loaded into a running kernel when needed and can be unloaded again. On Linux, modules are .ko files under /lib/modules/, loaded automatically when the hardware is detected, or by hand with modprobe. On Windows, kernel-mode drivers are .sys files, mostly in C:\Windows\System32\drivers. On Linux, lsmod lists what is loaded (illustrative output):

lsmod | head -5

Module                  Size  Used by
iwlmvm                798720  0
iwlwifi               598016  1 iwlmvm
snd_hda_intel          61440  3
nvme                   61440  3

Here iwlwifi and iwlmvm drive an Intel Wi-Fi adapter, snd_hda_intel the sound chip and nvme the SSD. The Used by column counts current users and names the modules that depend on each one.

Why a bad driver can crash the system

A loaded module or driver runs with exactly the same rights as the kernel. It can read any memory, and a bug in it can corrupt any kernel data. That is why drivers are the most common cause of stop errors, and why a driver is also a favorite target for attackers: malicious code in kernel mode can hide from antivirus software. A well-known example: in July 2024 a faulty content update, read by a security product's kernel-mode driver, made about 8.5 million Windows computers crash at every boot, until technicians deleted the bad file by hand in safe mode or the recovery environment.

Operating systems defend against bad drivers in several ways:

• Driver signing. 64-bit Windows loads a kernel-mode driver only if it carries a valid digital signature. Since Windows 10 version 1607, new kernel-mode drivers must also be signed by Microsoft through its Hardware Dev Center; with Secure Boot on, the older exceptions for cross-signed drivers no longer apply. Linux distributions with Secure Boot likewise require signed modules, which is why a third-party graphics driver may ask you to enroll a key at the next boot. See Signed drivers and Plug and Play.
• User-mode drivers. Where speed allows, drivers run in user mode instead: Windows has a User-Mode Driver Framework (UMDF) used for many USB devices, sensors and phones; printer drivers also run in user mode, in the print spooler; and Apple has been replacing kernel extensions with DriverKit extensions that run in user space. A crash then takes down the driver, not the system.
• Rollback. When a stop error starts right after a driver update, rolling back the driver in Device Manager is usually the quickest fix; see Updates, Patches and Drivers and OS Troubleshooting.