Operational Services: NTP, SNMP and Syslog

Why these three services

A network that forwards packets but tells you nothing about itself is impossible to run. A network operations center (NOC) needs to know that something is wrong (preferably before users call), what happened, and when, to the second, on every device involved. Three old, simple and universally supported protocols provide the foundation:

• NTP — the same clock everywhere;
• SNMP — numbers and states, polled or pushed, for graphs and alarms;
• Syslog — the devices' own event messages.

All three are configured on nearly every router, switch, firewall and server you will meet, and they appear in both the CCNA and Network+ objectives.

NTP: one clock for the whole network

The Network Time Protocol (NTP) synchronizes the clocks of devices with a reliable time source over the network. It uses User Datagram Protocol (UDP) port 123, and it is accurate enough — typically within a few milliseconds on a local area network (LAN) — that you can put events from dozens of devices in the right order. The current version, NTPv4, is defined in RFC 5905.

Why accurate time matters

• Log correlation. A link flap on a switch, an Open Shortest Path First (OSPF) neighbor going down on a router and a burst of errors on a firewall are one incident — but only if their timestamps agree. With clocks minutes apart, you are guessing.
• Certificates. Transport Layer Security (TLS) certificates are valid between two dates. A device whose clock is wrong rejects valid certificates or accepts expired ones, and HTTP Secure (HTTPS) management or virtual private network (VPN) connections fail.
• Authentication. Kerberos, used by Active Directory, rejects requests whose clock differs from the server by more than five minutes by default. One-time codes from an authenticator app depend on time too.
• Evidence. In a security investigation, a log line is only as good as its timestamp.

Strata: distance from the reference

NTP is organized as a hierarchy. Each level is a stratum, a number that says how many steps a clock is away from a reference clock.

• Stratum 0 is the reference itself: a Global Positioning System (GPS) receiver, an atomic clock or a radio time signal. It is not on the network; it is wired to a server.
• Stratum 1 servers are directly connected to a stratum 0 source.
• Stratum 2 devices synchronize from stratum 1, stratum 3 from stratum 2, and so on up to 15. Stratum 16 means "not synchronized".

A lower stratum does not automatically mean a more accurate clock, but it does mean fewer hops that can each add error. A typical design: two or three internal NTP servers or core routers sync with reliable outside sources, and every other device in the network syncs with those internal servers.

NTP on Cisco IOS

A Cisco device acts as an NTP client as soon as you configure a server. Configure at least two, so a single failure does not leave the device drifting.

R1(config)# ntp server 192.0.2.123 prefer
R1(config)# ntp server 198.51.100.123
R1(config)# ntp source Loopback0

• ntp server 192.0.2.123 prefer — synchronize with this server, and prefer it when several are equally good.
• ntp source Loopback0 — send NTP requests from the loopback address, which never goes down with a physical link and stays reachable as long as any path to the router works, and is easy to allow in firewall rules on the NTP servers.

A router can also serve time to others: once it is synchronized it answers NTP clients automatically, and ntp master makes it an authoritative source based on its own clock — only for labs and isolated networks. NTP can be authenticated with keys (ntp authenticate, ntp authentication-key, ntp trusted-key) so that nobody can feed your devices a false time.

R1# show ntp status
Clock is synchronized, stratum 2, reference is 192.0.2.123
nominal freq is 250.0000 Hz, actual freq is 250.0003 Hz, precision is 2**10
ntp uptime is 1503600 (1/100 of seconds), resolution is 4000
reference time is EE67516F.3B645A1C (09:14:23.232 UTC Wed Sep 30 2026)
clock offset is -0.4521 msec, root delay is 12.87 msec
root dispersion is 23.15 msec, peer dispersion is 1.02 msec
loopfilter state is 'CTRL' (Normal Controlled Loop), drift is -0.000001201 s/s
system poll interval is 64, last update was 38 sec ago.

R1# show ntp associations

  address         ref clock       st   when   poll reach  delay  offset   disp
*~192.0.2.123     .GPS.            1     38     64   377 12.870  -0.452  1.020
+~198.51.100.123  .PPS.            1     12     64   377 15.210   0.318  1.254
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

• Clock is synchronized, stratum 2 — R1 syncs from a stratum 1 server, so it is stratum 2 itself. "Clock is unsynchronized" is the first thing to look for in a ticket about wrong timestamps.
• * marks the server R1 is actually following (sys.peer), + a good candidate.
• st is that server's stratum; reach is an octal record of the last eight polls: 377 means all eight answered.
• offset is the time difference in milliseconds — well below a millisecond here.

UTC and time zones in logs

NTP itself always carries Coordinated Universal Time (UTC); time zones are only a matter of display. Many operations teams keep every device on UTC: logs from sites in different countries line up without conversion, and there are no gaps or duplicate hours when daylight saving time starts or ends. If you do use local time (clock timezone), always show the zone in the timestamp, as in the syslog configuration below.

SNMP: monitoring devices

Manager and agent

The Simple Network Management Protocol (SNMP) lets a central monitoring system read, and in principle change, values on network devices. It has two roles:

• The agent is software on the managed device — a router, switch, printer, uninterruptible power supply (UPS) or server. It knows the device's values and answers questions about them.
• The manager, part of a network management system (NMS), asks the questions, stores the answers, draws graphs and raises alarms.

Information flows in two ways:

• Polling: the manager sends Get, GetNext or GetBulk requests to the agent on UDP port 161, typically every one to five minutes, and the agent responds. A Set request changes a value, though few networks allow writing over SNMP today.
• Notifications: the agent itself sends a message to the manager on UDP port 162 when something happens, such as an interface going down. An SNMP trap is sent once and not acknowledged — if the packet is lost, the event is lost. An inform (SNMPv2c and v3) is acknowledged by the manager and resent until it is.

Polling tells you the state at regular moments and is good for trends such as bandwidth; notifications tell you about events right away. Monitoring systems use both.

MIB and OIDs

Every value an agent can report has a unique name in a worldwide tree, the object identifier (OID): a series of numbers separated by dots. A Management Information Base (MIB) is a text file that describes a part of that tree: which objects exist, their OIDs, their data types and what they mean. The manager loads MIB files so it can show "ifInOctets" instead of a row of numbers.

The prefix 1.3.6.1.2.1 is the standard mib-2 branch that every vendor implements; vendors put their own objects under 1.3.6.1.4.1 (enterprises), followed by a number assigned to the vendor. The final .0 in sysUpTime.0 means "the single instance of this object"; for per-interface objects the last number is the interface index (ifIndex).

A counter such as ifInOctets only ever goes up. A monitoring system computes bandwidth from two samples: the difference in bytes, times eight, divided by the seconds between them. A 32-bit counter wraps back to zero after 4,294,967,295; at 1 Gbit/s that happens in about 34 seconds, which is why fast interfaces are polled with the 64-bit "HC" (high capacity) counters. The calculation, including one wrap:

COUNTER_MAX = 2**32          # ifInOctets is a 32-bit counter

def uptime(ticks):           # sysUpTime counts hundredths of a second
    seconds = ticks // 100
    days, rest = divmod(seconds, 86_400)
    hours, rest = divmod(rest, 3600)
    minutes, secs = divmod(rest, 60)
    return f"{days} days, {hours}:{minutes:02}:{secs:02}"

def octet_delta(old, new):
    if new >= old:
        return new - old
    return COUNTER_MAX - old + new   # the counter wrapped past 2**32 - 1

print("sysUpTime.0 =", uptime(123_456_789))

# ifInOctets for ifIndex 1, polled 300 seconds apart
first, second, interval = 4_294_000_000, 1_205_032_704, 300
octets = octet_delta(first, second)
bps = octets * 8 / interval
print(f"Received {octets:,} octets in {interval} s")
print(f"Average inbound rate: {bps / 1_000_000:.2f} Mbit/s")
print(f"Utilization of a 100 Mbit/s link: {bps / 100_000_000:.1%}")

sysUpTime.0 = 14 days, 6:56:07
Received 1,206,000,000 octets in 300 s
Average inbound rate: 32.16 Mbit/s
Utilization of a 100 Mbit/s link: 32.2%

From a Linux host with the Net-SNMP tools you can ask a device yourself. Here the credentials come from environment variables, so they do not end up in your shell history — though they are still visible in the process list while the command runs. For regular use, put them in ~/.snmp/snmp.conf (readable only by you, chmod 600):

snmpget -v3 -l authPriv -u nocmon -a SHA -A "$SNMP_AUTH" -x AES -X "$SNMP_PRIV" 10.0.50.1 1.3.6.1.2.1.1.3.0

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (123456789) 14 days, 6:56:07.89

SNMP versions

A community string is a shared password such as the infamous defaults public (read-only) and private (read-write). It travels unencrypted in every packet, so anyone who can capture traffic can read it. SNMPv3 replaces communities with users and offers three security levels: noAuthNoPriv (username only), authNoPriv (messages are authenticated, for example with SHA) and authPriv (authenticated and encrypted, for example with AES). Use authPriv wherever the devices support it.

SNMPv3 on Cisco IOS

R1(config)# snmp-server group NOC-RO v3 priv
R1(config)# snmp-server user nocmon NOC-RO v3 auth sha Ex4mple-Auth-Key priv aes 128 Ex4mple-Priv-Key
R1(config)# snmp-server host 10.0.50.20 version 3 priv nocmon
R1(config)# snmp-server enable traps snmp linkdown linkup

• snmp-server group NOC-RO v3 priv — a group whose members must use authPriv; without a write view the group is read-only.
• snmp-server user nocmon NOC-RO v3 auth sha … priv aes 128 … — user nocmon in that group, with a SHA authentication key and an AES-128 encryption key (the two keys here are examples; use long, unique secrets). SNMPv3 users are not shown in the running configuration; check them with show snmp user.
• snmp-server host 10.0.50.20 version 3 priv nocmon — send notifications to the NMS at 10.0.50.20, as that user.
• snmp-server enable traps snmp linkdown linkup — which events to send; here, interfaces going down and up.

Monitoring systems

In practice you rarely talk SNMP by hand. A monitoring system polls every device on a schedule, stores the values as time series, receives traps and syslog, and turns it all into:

• Dashboards — graphs of bandwidth, central processing unit (CPU) load, memory, errors and up/down status, per device and per site;
• Thresholds and alerts — "interface utilization above 90% for 10 minutes", "device not answering for 3 polls", sent to a ticket system, chat or on-call phone;
• History — "was this link already this busy last month?", the basis of capacity planning.

Examples of such systems are Zabbix, LibreNMS and PRTG; many organizations use one of them or a commercial platform from their network vendor. Newer devices can also stream values continuously (model-driven telemetry) instead of waiting to be polled — see Network Automation.

Syslog: the devices' own story

Syslog is the standard way for devices and servers to send log messages — interface changes, logins, configuration changes, protocol events, errors — to a central syslog server. Classic syslog uses UDP port 514. The classic BSD format, which Cisco IOS sends by default, is documented in RFC 3164; RFC 5424 defines a newer, structured format. Where messages must not be lost, syslog can also run over TCP, and where they must not be read on the way, syslog over TLS uses TCP port 6514 (RFC 5425).

A device keeps only a small buffer of recent messages in memory, and loses it on reboot — exactly when you need it. A central server keeps everything, for all devices, searchable in one place, and out of reach of an attacker who wipes a device's logs.

Facility and severity

Every syslog message carries two numbers. The facility says what kind of process sent it: kernel, mail, authentication, or one of the eight local0–local7 facilities (16–23) left free for other uses; Cisco devices use local7 by default. The severity level says how important the message is, from 0 (most severe) to 7:

A common mnemonic for the order 0 to 7: Every Awesome Cisco Engineer Will Need Ice cream Daily. On the wire both numbers are combined into one priority value, PRI = facility × 8 + severity, written in angle brackets at the start of the message (this <PRI> header is the same in RFC 3164 and RFC 5424): local7 (23) and notice (5) give <189>.

Reading a Cisco log line

Sep 30 09:14:23.232 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
Sep 30 09:14:24.232 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

• Sep 30 09:14:23.232 UTC — the timestamp, with milliseconds and the time zone. A timestamp that starts with an asterisk (*Sep 30 …) means the clock is not synchronized: do not trust it.
• %LINK-3-UPDOWN — Cisco's FACILITY-SEVERITY-MNEMONIC code: the part of the software that raised it (LINK — the physical interface), the severity (3, error) and a short name for the event. Cisco's "facility" here is not the same as the syslog facility.
• The rest is the human-readable text. Together, the two lines read: the interface lost its physical link (layer 1, severity 3), and one second later the line protocol (layer 2) went down too (severity 5).

Syslog on Cisco IOS

R1(config)# service timestamps log datetime msec show-timezone
R1(config)# logging host 10.0.50.30
R1(config)# logging trap informational
R1(config)# logging source-interface Loopback0
R1(config)# logging buffered 16384 informational

• service timestamps log datetime msec show-timezone — stamp messages with date and time, milliseconds and the time zone (instead of the uptime of the device).
• logging host 10.0.50.30 — send messages to this syslog server (UDP 514).
• logging trap informational — send levels 0 to 6 to the server (informational is also the default), leaving out debugging.
• logging source-interface Loopback0 — send from a stable address, so the server always sees the same sender.
• logging buffered 16384 informational — also keep recent messages in a 16 KB memory buffer, readable with show logging.

Parsing a syslog message in Python

Log tooling does exactly this at scale: split a message into its fields so it can be filtered and counted. The message below is in the classic BSD format of RFC 3164, as Cisco IOS sends it, starting with the <PRI> header. The regular expression below uses the re module; the priority value is split with divmod (see Operators and Expressions).

import re

SEVERITIES = ["emergency", "alert", "critical", "error",
              "warning", "notice", "informational", "debugging"]

line = ("<189>42: Sep 30 09:14:24.232 UTC: %LINEPROTO-5-UPDOWN: "
        "Line protocol on Interface GigabitEthernet0/1, changed state to down")

pattern = re.compile(r"<(\d+)>.*?%([A-Z0-9_]+)-(\d)-([A-Z0-9_]+): (.*)")
pri, source, level, mnemonic, text = pattern.match(line).groups()

facility, severity = divmod(int(pri), 8)
print("Syslog facility:", facility, "(local7)" if facility == 23 else "")
print("Severity:", severity, SEVERITIES[severity])
print("Cisco facility:", source, "| mnemonic:", mnemonic)
print("Severity in text matches PRI:", int(level) == severity)
print("Message:", text)

trap_level = SEVERITIES.index("informational")    # logging trap informational
sent = [name for number, name in enumerate(SEVERITIES) if number <= trap_level]
print("Sent to the syslog server:", ", ".join(sent))

Syslog facility: 23 (local7)
Severity: 5 notice
Cisco facility: LINEPROTO | mnemonic: UPDOWN
Severity in text matches PRI: True
Message: Line protocol on Interface GigabitEthernet0/1, changed state to down
Sent to the syslog server: emergency, alert, critical, error, warning, notice, informational

How they work together in a NOC

A typical incident shows how they depend on each other:

1. A fiber between two buildings is damaged. The switch sends an SNMP linkDown trap; seconds later the NMS also sees the switch's uplink stop answering polls, and an alert opens a ticket.
2. The engineer searches the syslog server for the time of the alert and finds %LINK-3-UPDOWN on the switch, an OSPF adjacency change on the router at the other end, and a routing change on the core.
3. Because NTP keeps every clock in sync, those messages line up to the millisecond: first the link, then OSPF, then the traffic shift. Without NTP the same messages would sit minutes apart, and the cause could look like the effect.
4. Afterward, the SNMP bandwidth graphs show whether the backup path had enough capacity while the fiber was down.