Debugging and Testing

What is a bug?

A bug is a mistake in a program that makes it behave differently from what was intended: it crashes, gives a wrong answer, or does something unexpected. Debugging is the work of finding the cause of a bug and removing it; testing is the work of checking, as automatically as possible, that the program does what it should.

The word is older than computers — engineers talked about "bugs" in machines in the 19th century. A famous moment came in 1947, when operators of the Harvard Mark II computer found a moth stuck in a relay and taped it into their logbook as the "first actual case of bug being found". The joke only works because the word was already in use.

Testers split "bug" into error, defect and failure; see Errors, defects and failures.

Bugs are normal. Every programmer writes them, every day. What separates experienced programmers is not that they make fewer mistakes, but that they find them faster — with a method, the right tools, and tests that catch mistakes early.

Kinds of errors

Errors fall into three groups, depending on when they show up. The later an error appears, the harder it usually is to find.

Syntax errors

A syntax error breaks the grammar rules of the language: a missing comma, an unclosed bracket, a misspelled keyword. The compiler or interpreter cannot even understand the program, so nothing runs — not even the lines before the mistake:

print("Calculating the total...")
prices = [2.50, 3.75, 1.20]
print("Total:" sum(prices))

SyntaxError: invalid syntax. Perhaps you forgot a comma?

Note that the first line was never printed. Syntax errors are the easiest kind to fix: the message points to the line, and modern editors underline them while you type.

Runtime errors

A runtime error occurs while a syntactically correct program is running, when it attempts an operation that cannot be carried out: dividing by zero, opening a file that does not exist, using an index past the end of a list. In Python, a runtime error raises an exception; if nothing handles it, the program stops.

def average(values):
    return sum(values) / len(values)


print(average([18, 21, 24]))
print(average([]))  # no measurements today

21.0
ZeroDivisionError: division by zero

The first call works; the second crashes, because an empty list has length 0. Runtime errors often depend on the input, which is why a program can run fine for months and then fail on one unusual value.

Logic errors

A logic error is a mistake in the reasoning of a program: the code runs without any error message, but produces the wrong result. The computer does exactly what you wrote — just not what you meant.

def average_of_two(a, b):
    return a + b / 2  # meant: (a + b) / 2


print(average_of_two(18, 22))

29.0

The answer should be 20.0, but because division takes precedence over addition, the function computes 18 + 11. Nothing warns you. Logic errors are the most dangerous kind, because the only way to notice them is to know what the right answer is — which is exactly what tests are for.

Reading error messages

When a program crashes, the error message is your first and best clue — yet beginners often skip it. A Python error report, the traceback, contains three things:

• The error type and message, on the last line — for example ZeroDivisionError: division by zero. Read this first.
• The location: the file, line number and function where the error happened, with the line itself.
• The path that led there: every function call that was active, from the start of the program down to the failing line.

The line where a program crashes is not always the line that is wrong. In the average([]) example, the crash happens in average, but you could argue the real mistake is the caller passing an empty list. The traceback shows the whole chain, so you can decide where the fix belongs. The Python page on reading a traceback walks through one line by line. Other languages show the same information under a different name, such as a stack trace.

A systematic debugging method

Randomly changing code until the symptom disappears is slow and often creates new bugs. A method that works for any language and any size of bug:

1. Reproduce. Find the exact steps and input that make the bug appear, every time. A bug you cannot reproduce, you cannot confirm as fixed.
2. Isolate. Narrow down where it happens. Remove parts of the input, comment out code, or check values halfway through, until you know the smallest piece of code that goes wrong. Halving the search space each time is called a binary search for the bug.
3. Hypothesize. Form a concrete explanation: "the loop skips the last item because the range stops one too early".
4. Test the hypothesis. Check it with a print, a debugger or a small experiment. If it is wrong, go back to step 3 with what you learned.
5. Fix the cause, not the symptom. Adding if result == 29.0: result = 20.0 hides one case; correcting the formula fixes all of them.
6. Verify. Run the reproduction again, and the rest of your tests, to confirm the bug is gone and nothing else broke. Then add a test for this bug (see regression tests).

Debugging tools

Print debugging

The simplest tool is to print the values you are unsure about. This function should count the words of at least five letters, but returns 0. Python's f"{name=}" syntax prints both the expression and its value:

def count_long_words(text, min_length):
    count = 0
    for word in text.split():
        print(f"DEBUG {word=} {len(word)=}")
        if len(word) > min_length:
            count += 1
    return count


print(count_long_words("the quick brown fox", 5))

DEBUG word='the' len(word)=3
DEBUG word='quick' len(word)=5
DEBUG word='brown' len(word)=5
DEBUG word='fox' len(word)=3
0

The output shows that "quick" has length 5 and is still not counted — so the comparison must be wrong: > should be >=. Print debugging is quick and works everywhere, but remember to remove the prints afterward.

Logging

Logging is print debugging made permanent and configurable. Instead of print(), you write messages with a level — DEBUG, INFO, WARNING, ERROR, CRITICAL — and configure once which levels are shown and where they go (the screen, a file, a central log server). In production you show only warnings and errors; while debugging you switch on DEBUG, without touching the code.

import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.DEBUG,
    format="%(levelname)s %(funcName)s: %(message)s",
)


def apply_discount(price, percent):
    logging.debug("price=%s percent=%s", price, percent)
    discounted = price - price * percent / 100
    logging.debug("discounted=%s", discounted)
    return round(discounted, 2)


print(apply_discount(80.0, 15))

DEBUG apply_discount: price=80.0 percent=15
DEBUG apply_discount: discounted=68.0
68.0

Change level=logging.DEBUG to level=logging.WARNING and the two DEBUG lines disappear, while the messages stay in the code for the next time you need them. (By default, logging writes to standard error; stream=sys.stdout sends it to standard output here.)

Debuggers

A debugger is a tool that runs a program under your control: you can pause it, look at the values of its variables, and continue one line at a time. Every serious editor and IDE has one built in, usually with a graphical view of the variables.

You start by setting a breakpoint: a marker on a line of code where the debugger pauses the program before that line runs. In an editor you usually click in the margin next to the line number. While paused, you use these commands:

Python ships with a text-based debugger, pdb. The easiest way to use it is to put the built-in call breakpoint() on the line where you want to pause, and run the script as usual. You get a (Pdb) prompt where you can type the commands above. For the word-counting function, pausing at the comparison while word is "quick" would show:

(Pdb) p word
'quick'
(Pdb) p len(word), min_length
(5, 5)
(Pdb) p len(word) > min_length
False

A debugger shines when you do not yet know what to print: you can explore freely without editing and rerunning the program. Remove every breakpoint() call before you share your code.

Rubber duck debugging

Explain your code, line by line, out loud, to someone else — or to a rubber duck on your desk. The technique is named after a story in the book The Pragmatic Programmer. It sounds silly and works remarkably often: to explain what each line does, you have to stop assuming and actually check, and halfway through you hear yourself say "…and then this adds one to — oh." The duck never needs to answer.

Automated testing

Checking a program by hand — run it, type some input, look at the output — is slow, and nobody repeats it after every change. An automated test is code that calls your code with known input and checks the result against the expected answer. Once written, it runs in milliseconds, as often as you like.

Tests come at different sizes. A unit test checks one small unit of code, usually a single function or class, in isolation from the rest of the program. Integration tests check that several parts work together (your code and a real database, for example), and end-to-end tests drive the whole application the way a user would. Unit tests are the fastest and most precise, so a healthy project has many of them. The full set of test levels (component, integration, system and acceptance) and how they fit together is on Test Levels and Test Types.

Unit tests in Python

Python's standard library includes unittest. You write a class that inherits from unittest.TestCase; each method whose name starts with test is one test, and methods such as assertEqual compare the actual result with the expected one. A good test name says what behavior it checks:

import io
import unittest


def shipping_cost(order_total):
    """Orders of 50.00 or more ship free; smaller orders pay 4.95."""
    if order_total >= 50:
        return 0.0
    return 4.95


class ShippingCostTest(unittest.TestCase):
    def test_small_order_pays_shipping(self):
        self.assertEqual(shipping_cost(20.00), 4.95)

    def test_large_order_ships_free(self):
        self.assertEqual(shipping_cost(75.00), 0.0)

    def test_exactly_fifty_ships_free(self):
        self.assertEqual(shipping_cost(50.00), 0.0)


# Run the tests and print a short summary
suite = unittest.defaultTestLoader.loadTestsFromTestCase(ShippingCostTest)
result = unittest.TextTestRunner(stream=io.StringIO()).run(suite)
print("Tests run:", result.testsRun)
print("All passed:", result.wasSuccessful())

Tests run: 3
All passed: True

Notice the third test: the boundary value, exactly 50.00. Bugs love boundaries — > versus >=, the first or last item, an empty list — so test them explicitly. In a real project you do not write the last four lines; you put the tests in a file such as test_shipping.py and run python3 -m unittest, which finds and runs them and prints a report.

Many Python projects use the third-party tool pytest instead, where a test is a plain function with a plain assert statement:

# test_shipping.py — run with: pytest
from shipping import shipping_cost


def test_small_order_pays_shipping():
    assert shipping_cost(20.00) == 4.95


def test_exactly_fifty_ships_free():
    assert shipping_cost(50.00) == 0.0

pytest, with fixtures, parametrized tests and failure reports, is covered in depth on Unit Testing with pytest.

Other languages have the same idea under other names: JUnit for Java, Jest or Vitest for JavaScript, NUnit for C#. They all follow the pattern arrange (set up the input), act (call the code), assert (check the result).

Test-driven development

Test-driven development (TDD) turns the usual order around: you write a test for the next small piece of behavior before you write the code that implements it. You then work in a short, repeating cycle:

1. Red — write a test for behavior that does not exist yet, and run it. It must fail; that proves the test actually checks something.
2. Green — write the simplest code that makes the test pass. No more.
3. Refactor — clean up the code (better names, less repetition) while all tests stay green.

Suppose negative order totals should be rejected with a ValueError. In the red step you add this test to the class above and run the suite again, still with the old function:

import io
import unittest


def shipping_cost(order_total):
    if order_total >= 50:
        return 0.0
    return 4.95


class ShippingCostTest(unittest.TestCase):
    def test_exactly_fifty_ships_free(self):
        self.assertEqual(shipping_cost(50.00), 0.0)

    def test_negative_total_is_rejected(self):  # the new test
        with self.assertRaises(ValueError):
            shipping_cost(-5.00)


suite = unittest.defaultTestLoader.loadTestsFromTestCase(ShippingCostTest)
result = unittest.TextTestRunner(stream=io.StringIO()).run(suite)
print("Tests run:", result.testsRun)
for test, _ in result.failures + result.errors:
    print("FAILED:", test.id())
print("All passed:", result.wasSuccessful())

Tests run: 2
FAILED: __main__.ShippingCostTest.test_negative_total_is_rejected
All passed: False

Red, as expected. For green, you add the check to the function, and the whole suite passes again:

def shipping_cost(order_total):
    if order_total < 0:
        raise ValueError("order total cannot be negative")
    if order_total >= 50:
        return 0.0
    return 4.95

TDD forces you to decide what the code should do before deciding how, and it leaves a complete test suite behind as a side effect. Not every team uses it strictly, but the red-green-refactor rhythm is useful even when you write the test just after the code. A full TDD session, and the related practices BDD and ATDD, are on Test-Driven Development.

Regression tests

A regression is a bug in behavior that used to work, introduced by a later change. You fix a bug in March; in June someone refactors the function and the bug quietly returns. Regressions are common, because a change in one place can affect code far away.

The defense is a regression test: whenever you fix a bug, you add a test that reproduces it. The test fails before the fix and passes after it, and from then on it guards that bug forever. Tests do not need a framework; plain functions with assert work too:

def average(values):
    if not values:  # fix for the empty-list crash
        return 0.0
    return sum(values) / len(values)


def test_average_of_three_temperatures():
    assert average([18, 21, 24]) == 21.0


def test_empty_list_returns_zero():  # regression test for the crash
    assert average([]) == 0.0


tests = [test_average_of_three_temperatures, test_empty_list_returns_zero]
for test in tests:
    test()  # a failing assert would stop the program here
print(len(tests), "tests passed")

2 tests passed

Running the full test suite after every change — ideally automatically, each time code is pushed to a shared Git repository (continuous integration) — is what makes it safe to keep changing a program for years.

Writing testable code

Some code is easy to test and some is nearly impossible. The difference is mostly in how it is structured:

• Prefer pure functions. A function whose result depends only on its arguments, and that changes nothing else, is trivial to test: call it, compare the result. See Functions and Abstraction.
• Separate calculation from input and output. A function that reads a file, computes a total and prints it needs a file and captured output to test. Split it: one function computes the total from data, another does the reading and printing. Test the first thoroughly.
• Pass dependencies in. Code that looks up the current date or calls a web service inside itself behaves differently every run. Give it the date or the service as an argument, and a test can pass a fixed value or a stand-in object (see Test Doubles).
• Keep functions small. A function that does one thing needs a handful of tests; a function that does ten things needs hundreds of combinations.

These are the same qualities that make code easy to read and reuse. Testable code is, almost always, simply well-designed code.