Debugging C

How a C program fails

A Python or Java program that hits a bug usually stops with an exception and a traceback that names the file, the line and the call chain. C has no exceptions and no runtime that watches over the program (see Error Handling). When a C program breaks, one of three things happens:

• The operating system stops it. The program touched memory it may not use, and the kernel sends it a signal. The shell prints one line, such as Segmentation fault, and that is all.
• The program stops itself. A failed assert or a check inside the C library (such as glibc's double-free detection) calls abort(): the shell prints Aborted.
• Nothing visible happens. The program writes past the end of an array, reads freed memory or overflows an int, and carries on with wrong data. This is the most common case and the most dangerous one, because the damage shows up later, somewhere else, or never in testing.

Segmentation faults

A segmentation fault is the crash that follows when a program accesses memory that is not mapped into its address space or not allowed for that kind of access, such as reading through a null pointer or writing to a string literal; the kernel sends the signal SIGSEGV, which ends the program. The memory protection behind it is explained in Kernel and System Calls and Memory Management. Here a function adds up meter readings, but the array it gets is a null pointer because loading failed and nobody checked:

#include <stdio.h>

struct meter {
    const char *id;
    int kwh;
};

static int total_kwh(const struct meter *m, int count)
{
    int total = 0;
    for (int i = 0; i < count; i++) {
        total += m[i].kwh;
    }
    return total;
}

int main(void)
{
    struct meter *meters = NULL; /* loading failed, and nobody checked */
    printf("adding up 3 meters\n");
    printf("total: %d kWh\n", total_kwh(meters, 3));
    return 0;
}

adding up 3 meters
Segmentation fault

The output tells you the program got past its first printf and nothing more: no function name, no line. In a terminal you see the same one line, and the exit status encodes the signal:

$ ./meters
adding up 3 meters
Segmentation fault
$ echo $?
139

Exit status and signals

When a process is ended by a signal, the shell reports an exit status of 128 plus the signal number. That makes the number in a log or a monitoring alert readable:

assert and abort

An assert states something that must be true at that point. When it is false, glibc prints the file, line, function and the condition to standard error, then aborts. This is the one crash that explains itself:

#include <assert.h>
#include <stdio.h>

static int average(const int *watts, int count)
{
    assert(count > 0);
    int total = 0;
    for (int i = 0; i < count; i++) {
        total += watts[i];
    }
    return total / count;
}

int main(void)
{
    int watts[] = {120, 95, 130, 110};
    printf("average: %d W\n", average(watts, 4));
    printf("average: %d W\n", average(watts, 0));
    return 0;
}

average: 113 W
example: example.c:6: average: Assertion `count > 0' failed.
Aborted

Assertions are for programming errors, not for bad input or a missing file: a build with -DNDEBUG removes them completely. Return values and errno handle expected failures (see Error Handling).

Before the debugger

The cheapest bugs to fix are the ones the compiler finds. Build with -Wall -Wextra (and -Werror in your own projects) before you start any debugging session: an uninitialized variable, a missing return, a printf format that does not match its argument or an if (status = 1) is reported with the exact line, and many of them are exactly the "impossible" behavior you were about to chase. See Compiling and Running C for what each flag catches.

Print debugging

Adding print statements is still a perfectly good method (see Debugging and Testing). Two C-specific rules make it reliable:

• Print to standard error. fprintf(stderr, …) keeps debug lines out of the program's real output, and standard error is unbuffered, so the line is written before the next statement runs.
• Remember buffering. Standard output to a pipe or file is collected in a buffer and written later. If the program crashes, buffered printf output is lost, and you conclude it crashed earlier than it did. Use stderr, or call fflush(stdout) after each debug line (see Input, Output and Files).

#include <stdio.h>

#define DEBUG 1

int main(void)
{
    int watts[] = {120, 95, 130, 110};
    int total = 0;
    for (int i = 0; i < 4; i++) {
        total += watts[i];
        if (DEBUG) {
            fprintf(stderr, "debug %s:%d: i=%d total=%d\n",
                    __FILE__, __LINE__, i, total);
        }
    }
    printf("total: %d W\n", total);
    return 0;
}

debug example.c:13: i=0 total=120
debug example.c:13: i=1 total=215
debug example.c:13: i=2 total=345
debug example.c:13: i=3 total=455
total: 455 W

__FILE__ and __LINE__ are filled in by the preprocessor (see The Preprocessor and Build Tools). Print debugging falls short when you do not know where to look, when the bug disappears as soon as you add a print, or when the program crashes before your print. That is when a debugger pays off.

Debug information

Debug information is extra data that the compiler stores in a program or object file when you pass -g: which machine instructions belong to which source line, and the names, types and locations of functions, parameters and variables. A debugger needs it to talk about your source code instead of raw addresses. On Linux it is stored in the DWARF format, in separate sections of the executable that are not loaded when the program runs normally.

cat > meters.c <<'EOF'
#include <stdio.h>
int main(void) { printf("meters: %d\n", 3); return 0; }
EOF
gcc -std=c17 -O0 -o plain meters.c
gcc -std=c17 -O0 -g -o debug meters.c
for f in plain debug; do
    printf '%-6s %s\n' "$f" "$(file -b "$f" | grep -o 'with debug_info' || echo 'no debug info')"
done
./debug

plain  no debug info
debug  with debug_info
meters: 3

Debug information does not change what the program does; it makes the file bigger. Release builds are often stripped (strip removes symbols and debug sections), and distributions ship the debug information separately (Debian's -dbgsym packages, for example), so that a crash on a production server can still be analyzed later.

Optimization and debugging

Optimization rearranges code: variables live only in registers or disappear, lines run out of order, small functions are inlined into their callers. A debugger then shows a program that no longer matches the source line by line. So debug with -O0 (GCC also offers -Og, optimization that keeps debugging usable). The difference is easy to see. The average program from Breakpoints and stepping, built with -O2 -g, has no average function left to stop in, because the compiler inlined it into main:

$ gcc -std=c17 -O2 -g -o average average.c
$ gdb -q -batch -ex 'break average' -ex run ./average
Function "average" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
average: 83 W
[Inferior 1 (process 12345) exited normally]

Some bugs appear only in the optimized build. That is nearly always undefined behavior that the optimizer exploited, not a compiler bug; a sanitizer build (below) usually finds it.

Stepping through code with GDB

GDB, the GNU Debugger, is the standard command-line debugger for C on Linux and other Unix-like systems: it starts a program (or attaches to a running one, or opens a core dump), stops it at chosen points, and lets you inspect and change its variables, memory and call stack. It is the tool behind most graphical debuggers in editors on Linux. The recordings on this page use GDB 13.1; LLDB, the debugger from the LLVM project, does the same job with slightly different commands. The general idea of a debugger is in Debugging and Testing.

Finding the crashing line

Rebuild the crashing program from the start of this page with -g, saved as meters.c, and run it under GDB. run starts it; when the signal arrives, GDB stops the program at the exact instruction and shows the source line:

$ gcc -std=c17 -Wall -Wextra -O0 -g -o meters meters.c
$ gdb -q ./meters
Reading symbols from ./meters...
(gdb) run
Starting program: /home/amara/meters/meters
adding up 3 meters

Program received signal SIGSEGV, Segmentation fault.
0x00005555555507fc in total_kwh (m=0x0, count=3) at meters.c:12
12	        total += m[i].kwh;
(gdb) bt
#0  0x00005555555507fc in total_kwh (m=0x0, count=3) at meters.c:12
#1  0x0000555555550858 in main () at meters.c:21
(gdb) print m
$1 = (const struct meter *) 0x0
(gdb) info locals
i = 0
total = 0
(gdb) frame 1
#1  0x0000555555550858 in main () at meters.c:21
21	    printf("total: %d kWh\n", total_kwh(meters, 3));
(gdb) print meters
$2 = (struct meter *) 0x0
(gdb) kill
Kill the program being debugged? (y or n) y
[Inferior 1 (process 12345) killed]
(gdb) quit

In six commands the mystery is solved: the crash is on line 12, in total_kwh, called from line 21 of main, and the pointer m is null (0x0) with i = 0, so the very first access failed. The fix belongs where the pointer came from: check the result of loading before using it.

The list printed by bt is a backtrace: the chain of function calls that are active at the moment the program stopped, innermost first, one line per stack frame, with the function's arguments and the file and line where each frame is. It is the C equivalent of a Python traceback, read in the opposite direction (frame #0 is where it stopped). Without -g the same backtrace only shows total_kwh () and main (): no arguments, no files, no lines.

Breakpoints and stepping

Not every bug crashes. This program computes the average power of four readings and prints a value that is clearly too low (the readings are around 100 W, the right average is 113 W):

#include <stdio.h>

static int average(const int *watts, int count)
{
    int total = 0;
    for (int i = 1; i < count; i++) {
        total += watts[i];
    }
    return total / count;
}

int main(void)
{
    int watts[] = {120, 95, 130, 110};
    int avg = average(watts, 4);
    printf("average: %d W\n", avg);
    return 0;
}

average: 83 W

A breakpoint stops the program at a function or line; from there you step through it and look at the values:

$ gdb -q ./average
Reading symbols from ./average...
(gdb) break average
Breakpoint 1 at 0x760: file average.c, line 5.
(gdb) run
Starting program: /home/amara/meters/average

Breakpoint 1, average (watts=0x7fffffffece8, count=4) at average.c:5
5	    int total = 0;
(gdb) print count
$1 = 4
(gdb) print *watts@count
$2 = {120, 95, 130, 110}
(gdb) next
6	    for (int i = 1; i < count; i++) {
(gdb) next
7	        total += watts[i];
(gdb) print i
$3 = 1

print *watts@count shows count elements starting at the pointer, which a plain print watts cannot do, because inside the function watts is only a pointer (see Arrays and Pointer Arithmetic). The data is right. But the first time the loop body runs, i is already 1: the loop skips the first reading.

Watchpoints

A watchpoint stops the program whenever the value of a variable or memory location changes (or, with rwatch and awatch, when it is read), wherever in the program that happens. It answers the question "who changes this value?", which breakpoints cannot answer when you do not know where to put them. Continuing the session above:

(gdb) watch total
Hardware watchpoint 2: total
(gdb) continue
Continuing.

Hardware watchpoint 2: total

Old value = 0
New value = 95
average (watts=0x7fffffffece8, count=4) at average.c:6
6	    for (int i = 1; i < count; i++) {
(gdb) continue
Continuing.

Hardware watchpoint 2: total

Old value = 95
New value = 225
average (watts=0x7fffffffece8, count=4) at average.c:6
6	    for (int i = 1; i < count; i++) {
(gdb) delete
Delete all breakpoints? (y or n) y
(gdb) finish
Run till exit from #0  average (watts=0x7fffffffece8, count=4) at average.c:6
0x00005555555507f4 in main () at average.c:15
15	    int avg = average(watts, 4);
Value returned is $4 = 83
(gdb) continue
Continuing.
average: 83 W
[Inferior 1 (process 12345) exited normally]
(gdb) quit

The total goes from 0 straight to 95, the second reading: 120 never gets added. The fix is int i = 0, a classic off-by-one error. "Hardware watchpoint" means the processor's debug registers do the watching, so the program runs at full speed until the value changes. A watchpoint on a local variable ends when its function returns.

The commands you need most

Pressing Enter on an empty line repeats the last command, which makes stepping with next quick. GDB can also attach to a process that is already running (gdb -p PID), which pauses it: never do that to a production service without agreeing on it first. For a program on another machine or a microcontroller, GDB connects remotely, through gdbserver or a hardware debug probe; firmware debugging is covered in Firmware and Embedded Programming.

Core dumps: debugging after the crash

A crash on a server or at a customer cannot be rerun under GDB on request. A core dump solves that: the kernel writes the memory and registers of the crashing process to a file, and GDB opens that file together with the executable as if the program had just stopped. Whether and where core dumps are written depends on the system: the shell's ulimit -c (often 0, meaning off) and the kernel setting /proc/sys/kernel/core_pattern. On systems with systemd-coredump, coredumpctl list and coredumpctl gdb find and open them.

$ ulimit -c unlimited
$ ./meters
adding up 3 meters
Segmentation fault (core dumped)
$ gdb -q ./meters core
Reading symbols from ./meters...
[New LWP 12345]
Core was generated by `./meters'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00005556517f07fc in total_kwh (m=0x0, count=3) at meters.c:12
12	        total += m[i].kwh;
(gdb) bt
#0  0x00005556517f07fc in total_kwh (m=0x0, count=3) at meters.c:12
#1  0x00005556517f0858 in main () at meters.c:21

The core dump only helps together with the exact executable that crashed, ideally with its debug information. Keep build artifacts (or the debug symbols) for every released version.

Sanitizers: catching silent memory bugs

The worst C bugs do not crash. This program writes one element past the end of a heap array; without any checking it runs to the end and exits with status 0:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    int count = 4;
    int *watts = malloc(count * sizeof *watts);
    if (watts == NULL) {
        return EXIT_FAILURE;
    }
    for (int i = 0; i <= count; i++) {   /* <= is one too many */
        watts[i] = 100 + i;
    }
    printf("last reading: %d W\n", watts[count - 1]);
    free(watts);
    return 0;
}

last reading: 103 W

The fifth write landed in memory that belongs to the allocator or to another object. Today nothing visible happens; with a different allocation pattern, input or compiler, the same line corrupts data or crashes somewhere far away. This is why "it works on my machine" proves very little in C.

A sanitizer is a run-time checker that the compiler builds into a program: with an option such as -fsanitize=address, GCC and Clang add checks around memory accesses and operations, and a support library reports the first violation with its location and, for most kinds, stops the program. The program is slower and uses more memory, so sanitizers are for development, tests and fuzzing, not for production builds.

AddressSanitizer

AddressSanitizer (ASan) is the sanitizer for memory errors, enabled with -fsanitize=address: it detects out-of-bounds reads and writes on the heap, the stack and global variables, use of freed memory, double frees and (through its built-in LeakSanitizer) memory leaks. It was developed at Google, described in a USENIX paper in 2012, and has been part of GCC since version 4.8 and of Clang. The same program, built with sanitizers, stops at the bad write:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    int count = 4;
    int *watts = malloc(count * sizeof *watts);
    if (watts == NULL) {
        return EXIT_FAILURE;
    }
    for (int i = 0; i <= count; i++) {   /* <= is one too many */
        watts[i] = 100 + i;
    }
    printf("last reading: %d W\n", watts[count - 1]);
    free(watts);
    return 0;
}

SUMMARY: AddressSanitizer: heap-buffer-overflow example.c:12 in main

The full report says much more. Read it from the top: what kind of error, what access (write or read, how many bytes), where in the code, and where the memory came from:

$ gcc -std=c17 -Wall -Wextra -O0 -g -fsanitize=address,undefined -o heap heap.c
$ ./heap
=================================================================
==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7fff7b2007c0 at pc 0x555591f21108 bp 0x7ffffac1c6b0 sp 0x7ffffac1c6c8
WRITE of size 4 at 0x7fff7b2007c0 thread T0
    #0 0x555591f21104 in main heap.c:12
    …

0x7fff7b2007c0 is located 0 bytes to the right of 16-byte region [0x7fff7b2007b0,0x7fff7b2007c0)
allocated by thread T0 here:
    #0 0x7fff7e9bb734 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x555591f20ff0 in main heap.c:7
    …

SUMMARY: AddressSanitizer: heap-buffer-overflow heap.c:12 in main
Shadow bytes around the buggy address:
    … (a map of the surrounding memory and its legend)
==12345==ABORTING
$ echo $?
1

"4 bytes, 0 bytes to the right of a 16-byte region allocated on line 7": the program wrote one int directly after an array of four. The frames below main (__libc_start_main and friends) are the C library's start-up code and can be ignored.

Some memory bugs are visible to the compiler. GCC 12, the version used for this page, includes -Wuse-after-free in -Wall, so the simplest case of reading freed memory does not even compile with -Werror:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    int *watts = malloc(3 * sizeof *watts);
    if (watts == NULL) {
        return EXIT_FAILURE;
    }
    watts[0] = 1234;
    free(watts);
    printf("first reading: %d W\n", watts[0]);
    return 0;
}

example.c:12:5: error: pointer ‘watts’ used after ‘free’ [-Werror=use-after-free]

Once the free moves into another function, the compiler no longer sees it, and only the sanitizer catches the bad read. Without the sanitizer this program prints whatever happens to be in the freed memory: the old value, garbage, or another object's data. ASan's full report names three places: the bad access, where the memory was freed and where it was allocated.

#include <stdio.h>
#include <stdlib.h>

struct batch {
    int *watts;
    int count;
};

static void close_batch(struct batch *b)
{
    free(b->watts);   /* frees the readings, but leaves the pointer */
}

int main(void)
{
    struct batch b = {malloc(3 * sizeof(int)), 3};
    if (b.watts == NULL) {
        return EXIT_FAILURE;
    }
    b.watts[0] = 1234;
    close_batch(&b);
    printf("first reading: %d W\n", b.watts[0]);
    return 0;
}

SUMMARY: AddressSanitizer: heap-use-after-free example.c:22 in main

Memory leaks are reported when the program ends. The program's own output comes first; then LeakSanitizer lists every block that was never freed, with the line that allocated it:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static char *copy_id(const char *id)
{
    char *copy = malloc(strlen(id) + 1);
    if (copy != NULL) {
        strcpy(copy, id);
    }
    return copy;
}

int main(void)
{
    for (int home = 1; home <= 3; home++) {
        char *id = copy_id("KHP1-000231");
        if (id != NULL) {
            printf("dongle %s\n", id);
        }
        /* missing: free(id); */
    }
    return 0;
}

dongle KHP1-000231
dongle KHP1-000231
dongle KHP1-000231
SUMMARY: AddressSanitizer: 36 byte(s) leaked in 3 allocation(s).

Three copies of 12 bytes (11 characters and the null terminator) were never freed. In a program that runs for a few milliseconds that is harmless; in firmware or a service that does this for every message, memory runs out after days or weeks. The bugs themselves are explained in Dynamic Memory.

UndefinedBehaviorSanitizer

UndefinedBehaviorSanitizer (UBSan), enabled with -fsanitize=undefined, checks operations whose result the C standard leaves undefined: signed integer overflow, division by zero, shifts by too many bits, out-of-range array indexes it can see, misaligned or null pointer use and more. By default it prints a runtime error line and lets the program continue, so one test run can show several problems. Here the yearly number of meter-hours for Kestrel's 400,000 meters overflows an int:

#include <stdio.h>

int main(void)
{
    int meters = 400000;
    int hours = 8760;
    int meter_hours = meters * hours;
    printf("meter-hours per year: %d\n", meter_hours);
    return 0;
}

example.c:7:9: runtime error: signed integer overflow: 400000 * 8760 cannot be represented in type 'int'
meter-hours per year: -790967296

Without the sanitizer the program prints the same negative number without any warning, at least today: signed overflow is undefined behavior, and an optimizing compiler may produce something else entirely (see Undefined Behavior and Safer C). The fix is a wider type, int64_t (see Variables, Types and Integers). UBSan also catches the division by zero that ARM64 hides:

#include <stdio.h>

int main(int argc, char **argv)
{
    (void)argv;
    int homes = argc - 1;   /* no arguments given: 0 homes */
    printf("per home: %d W\n", 1234 / homes);
    return 0;
}

example.c:7:37: runtime error: division by zero
per home: 0 W

Other tools

Valgrind

Valgrind runs an unmodified program on a simulated processor and checks every memory access; its default tool, Memcheck, finds most of the heap errors AddressSanitizer finds, plus uses of uninitialized memory, but not overflows of stack or global arrays. It needs no rebuild (only -g for line numbers), which makes it useful for programs you cannot recompile with sanitizers, but the program runs many times slower. The current release is 3.27.1 (May 2026, as of October 2026). It was not installed on the machine used for this page, so the output below is illustrative:

$ valgrind ./heap_plain
==12345== Memcheck, a memory error detector
==12345== Invalid write of size 4
==12345==    at 0x108A1C: main (heap.c:12)
==12345==  Address 0x4a8b050 is 0 bytes after a block of size 16 alloc'd
==12345==    at 0x48850C8: malloc (vg_replace_malloc.c:381)
==12345==    by 0x108983: main (heap.c:7)
…
==12345== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Do not combine the two: run Valgrind on a normal build, not on a sanitizer build.

Static analysis and strace

Two more tools fit in the same toolbox. GCC's -fanalyzer and Clang's --analyze examine the code without running it and follow paths through it, reporting double frees, use after free, leaks in helper functions and unchecked malloc results; how this fits with coding standards is in Undefined Behavior and Safer C. And strace ./billing lists every system call a program makes: when a program "does nothing" or fails without a useful message, strace shows the file it could not open or the network call it is waiting for, without any source code.

Fuzzing

Sanitizers only report bugs on paths that run. Fuzzing generates thousands of inputs per second to reach new paths, and a sanitizer build turns every hidden memory error into a reported crash. Code that parses untrusted input is the first candidate, such as the P1 telegram parser in the Kestrel Home dongle's firmware, which BUG-1060 showed to be fragile (see Smart Meters and the P1 Port). With libFuzzer, part of the LLVM project, a harness is one function that receives the generated bytes:

#include <stddef.h>
#include <stdint.h>

/* The function under test, from the parser's own source file. */
int p1_parse_telegram(const char *text, size_t length);

/* libFuzzer calls this with generated input, over and over. */
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
    p1_parse_telegram((const char *)data, size);
    return 0;   /* any crash or sanitizer report is a finding */
}

$ clang -g -O1 -fsanitize=fuzzer,address,undefined fuzz_p1.c p1_parser.c -o fuzz_p1
$ ./fuzz_p1 corpus/
…
==12345==ERROR: AddressSanitizer: stack-buffer-overflow on address …
…
SUMMARY: AddressSanitizer: stack-buffer-overflow p1_parser.c:41 in p1_parse_telegram
artifact_prefix='./'; Test unit written to ./crash-…

The saved crash input becomes a regression test once the bug is fixed. The function name and the finding above are made up for the example. The dongle's firmware is the manufacturer's code, not Kestrel's, so in the story this is a test Kestrel can ask the manufacturer to run, not one it runs itself.

Choosing a tool

Reporting a crash

Service desk and operations staff rarely debug C code themselves, but the evidence they collect decides whether a developer can. A useful ticket for a crashing C program contains:

• Which program and version: the package version or the firmware version, and the machine (architecture and operating system: a bug may only appear on ARM64 or only on x86-64).
• How it ended: the exit status or signal (139, Segmentation fault, Aborted), and the last lines it printed, including any message from glibc or an assert.
• Whether a core dump exists, and where it is, instead of attaching it (it may contain personal data).
• A backtrace if you can produce one (coredumpctl gdb, then bt).
• The input and the time: which file, request or telegram triggered it, and whether it happens every time.

readings-import 2.3.1 (Debian 12, ARM64) stopped at 02:31 with exit status 139
(Segmentation fault) in the nightly run; the journal shows no other error.
Core dump kept on the server (coredumpctl list, PID 12345); not attached.
Backtrace: #0 total_kwh (m=0x0, count=3) at meters.c:12, #1 main () at meters.c:21.
Reproducible: yes, with yesterday's input file (meters loaded: 0).

A null pointer in frame #0 and "meters loaded: 0" point a developer straight at the unchecked load, without anyone having to guess.