Reading Errors, Logs and Documentation

Read the whole message

An error message is a program's report that something it tried to do failed, usually with the reason. It is the cheapest clue you will ever get, and it is skipped surprisingly often: people see red text, close the window and start guessing. Read every word, and copy the exact text into the ticket.

Most error messages answer three questions, if you look for them:

ssh: connect to host 192.0.2.10 port 22: Connection refused

• Who complains? ssh, the client program on your machine.
• What did it try? To connect to host 192.0.2.10 on port 22.
• Why did it fail? "Connection refused": the host answered, but nothing accepted the connection on that port — the SSH service is not running, or a firewall actively rejected it. Compare "Connection timed out", which means no answer came back at all: a host that is down, a wrong address, or a firewall that silently drops traffic. Two words make the difference between "look at the server" and "look at the network path".

Small words carry big meaning. "Permission denied" is not "No such file or directory"; "certificate has expired" is not "certificate is not trusted". Error codes help too: an HTTP 503 means the server says it cannot handle the request right now, while a 404 means the server works but the page does not exist (see status codes). Search the code, not only the wording, because wording is often translated into the user's language and codes are not.

Stack traces

A stack trace is the list of function calls that were active when a program failed, printed together with the error. Each line names a function, a file and a line number; together they show the path the program took to the failure. Python calls its version a traceback and prints the error last; reading one line by line is explained on reading a traceback.

Java and many other languages print the error first, and add "Caused by" sections for errors that triggered other errors. This one is illustrative, from a billing job:

Exception in thread "main" java.lang.IllegalStateException: Could not load tariffs
    at com.example.billing.TariffLoader.load(TariffLoader.java:42)
    at com.example.billing.InvoiceJob.run(InvoiceJob.java:18)
    at com.example.billing.Main.main(Main.java:9)
Caused by: java.io.FileNotFoundException: /data/tariffs-2026.csv (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:152)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:106)
    at com.example.billing.TariffLoader.load(TariffLoader.java:37)
    ... 2 more

How to read it, even if you never write Java:

• Find the deepest "Caused by". The top line says what failed in general ("could not load tariffs"); the last "Caused by" usually says why: a file that does not exist.
• Find the first line from the organization's own code (here com.example.billing). Lines from the language runtime or libraries are rarely where the mistake is.
• Turn it into an operations question. You do not need to fix the code: does /data/tariffs-2026.csv exist, is the name right, did a transfer job fail? That is often enough to solve it, or to escalate with a precise note.

Finding the first error

When something breaks, logs often fill with hundreds of errors within seconds. The first error — the earliest error, or warning, in the time window where the problem started — is usually the one closest to the cause. The rest are often consequences: a database that is unreachable makes every login fail, and every failed login produces its own error. The last error on the screen is the most visible and usually the least informative.

Classic Unix tools find it quickly. This example builds a small portal log and then asks three questions: how many errors, which came first, and which messages repeat:

cat > portal.log <<'EOF'
2026-09-14 10:01:12 INFO  login ok user=k.jansen
2026-09-14 10:01:17 WARN  session store slow (2.8 s)
2026-09-14 10:01:19 ERROR session store timeout
2026-09-14 10:01:19 ERROR login failed: no session
2026-09-14 10:01:20 ERROR HTTP 500 /login
2026-09-14 10:01:24 ERROR login failed: no session
2026-09-14 10:01:25 ERROR HTTP 500 /login
2026-09-14 10:01:31 ERROR login failed: no session
2026-09-14 10:01:31 ERROR HTTP 500 /login
EOF
grep -c ERROR portal.log
grep -n -m 1 ERROR portal.log
grep ERROR portal.log | cut -d ' ' -f 4- | sort | uniq -c | sort -rn

7
3:2026-09-14 10:01:19 ERROR session store timeout
      3 login failed: no session
      3 HTTP 500 /login
      1 session store timeout

The count shows that the frequent messages — failed logins and HTTP 500s — are not the interesting ones: they are consequences. The single "session store timeout" came first, and the warning two seconds earlier ("session store slow") pointed at the same component. That is where to look. The commands themselves are explained on grep, find and regular expressions and pipes and redirection; the journal and the files in /var/log on logs and monitoring.

With structured logs, the same search becomes a filter on level and message; see using logs during an incident.

Correlating logs across systems

A user-facing problem often shows up in one system and is caused in another. Log correlation means putting events from several logs — web server, database, firewall, authentication — on one timeline, so you can see what happened just before the symptom and where. It depends on two things:

• Clocks that agree. Servers and network devices should synchronize their time with the Network Time Protocol; a server that is three minutes off makes cause and effect appear in the wrong order. See why accurate time matters.
• One time zone. Some systems log in Coordinated Universal Time (UTC), others in local time, which in the Netherlands is UTC+1 in winter and UTC+2 in summer. Convert everything to UTC before comparing; see UTC and time zones in logs.

Here the portal logs in Amsterdam local time and the session database in UTC. At first sight the database log looks irrelevant — its times are two hours "earlier". After conversion, the script looks at the two minutes before the first portal error:

from datetime import datetime, timedelta, timezone

# Portal log: local time (Amsterdam, summer time = UTC+2)
portal_log = """\
2026-09-14T10:00:40+02:00 INFO  login ok user=k.jansen
2026-09-14T10:01:17+02:00 WARN  session store slow (2.8 s)
2026-09-14T10:01:19+02:00 ERROR session store timeout
2026-09-14T10:01:19+02:00 ERROR login failed: no session
2026-09-14T10:01:20+02:00 ERROR HTTP 500 /login"""

# Session database log: UTC
db_log = """\
2026-09-14T07:55:00+00:00 INFO  checkpoint complete
2026-09-14T08:00:58+00:00 WARN  disk usage 97%
2026-09-14T08:01:15+00:00 ERROR could not extend file: no space left on device
2026-09-14T08:12:30+00:00 INFO  checkpoint complete"""


def parse(text):
    for line in text.splitlines():
        stamp, level, message = line.split(maxsplit=2)
        yield datetime.fromisoformat(stamp), level, message


first_error = next(e for e in parse(portal_log) if e[1] == "ERROR")
when = first_error[0]
print("First portal error:", first_error[2])
print("  local:", when.isoformat())
print("  UTC:  ", when.astimezone(timezone.utc).isoformat())

print("Database events in the 2 minutes before:")
for stamp, level, message in parse(db_log):
    if when - timedelta(minutes=2) <= stamp <= when:
        print(f"  {stamp:%H:%M:%S} UTC {level:5} {message}")

First portal error: session store timeout
  local: 2026-09-14T10:01:19+02:00
  UTC:   2026-09-14T08:01:19+00:00
Database events in the 2 minutes before:
  08:00:58 UTC WARN  disk usage 97%
  08:01:15 UTC ERROR could not extend file: no space left on device

The portal's timeout was a consequence: the session database had run out of disk space. In larger environments, a central log platform and a shared request identifier (a correlation ID passed from system to system) do this matching automatically — see centralized logging and monitoring and event management. How to put correlation IDs and consistent log levels into your own services' logs is on Logs.

Searching effectively

Almost every error message has been seen before. Finding the explanation is a skill:

• Search the exact text in quotes, minus anything specific to you: remove host names, user names, IP addresses, file paths and numbers. "could not extend file" "no space left on device" finds others; your server name finds nobody.
• Add the product and version. Behavior and fixes differ between versions; an answer for version 12 may be wrong for version 16.
• Search the error code where there is one; it is the same in every language.
• Start inside your organization. The knowledge base, known errors and earlier tickets know your environment; the Internet does not. See the knowledge base.
• Check the date of what you find, and prefer recent answers for products that change quickly.

Documentation and release notes

Reading documentation

Documentation comes in layers. On Linux, the manual pages and --help are on the system itself and match the installed version (see manual pages). Vendors publish product documentation, knowledge-base articles and troubleshooting guides; standards bodies publish specifications such as Requests for Comments (RFCs) for Internet protocols.

Read documentation with a question in mind, and read around the answer: the paragraph before often says "this applies only when…", and the paragraph after often says "note: requires a restart". Check that the page is for your version — documentation sites usually have a version selector at the top.

Release notes

Release notes describe what changed in a new version of software: new features, fixed bugs, known issues, deprecated functions and breaking changes that need action. They are short, and they answer two very practical questions:

• Before an update: what will this change, and what must I do first? A line such as "TLS 1.0 support removed" can break old clients.
• After an update, when something breaks: is this a known issue, and is there a workaround or a fixed version?

When a problem starts right after an update, the release notes of that update — especially "Known issues" — are often the fastest route to the answer. How releases are planned and communicated is on release and deployment management.

Judging sources

Not every answer you find deserves the same trust. A primary source is the origin of the information: the vendor's own documentation and release notes, the standard itself, the source code, or your own test. Secondary sources — blog posts, forum answers, video tutorials, summaries — explain or repeat it, sometimes very well, sometimes outdated or wrong.

Questions that help you judge any source:

• Date and version: is it about the version you run?
• Does it explain why? An answer that explains the cause is easier to check than "this worked for me".
• Does it match your symptoms, including the exact error text, or only roughly?
• What does it change, and can you undo it? Read every command before you run it.
• Can you confirm it in a primary source or a test environment?

AI answers

AI chat assistants are quick at explaining an error message or suggesting where to look, and they are often right. They can also produce a confident, well-formatted answer that is wrong — a command option that does not exist, a setting from another product, a hallucination. Treat an AI answer like an anonymous forum post: a lead to check against a primary source and to test safely, never a fact to run on production. What to check and what never to paste into such tools is on using AI at work.