MQTT and IoT Messaging

What MQTT is

A sensor that measures something every ten seconds has to get that value somewhere. It could wait until a server asks (polling), or it could send the value itself when it has one. Most IoT devices do the second: they sit behind home routers and mobile networks that do not accept incoming connections, they sleep to save power, and their links drop without warning. A small protocol in which the device opens one outgoing connection and pushes messages over it fits that world well.

MQTT is such a protocol: a lightweight messaging protocol for publish-subscribe in which clients send messages to a central server, the broker, under a topic name, and receive messages for the topics they subscribed to. It runs over TCP, on the registered port numbers 1883 (unencrypted) and 8883 (TLS). The name once stood for MQ Telemetry Transport; today the standard simply calls it MQTT. Two versions are in use:

• MQTT 3.1.1, an OASIS standard since October 2014 and also published as ISO/IEC 20922:2016. Many small devices and libraries still use it, and it is the default of the Mosquitto command-line clients.
• MQTT 5.0, an OASIS standard since March 2019. It adds reason codes on every acknowledgment, session and message expiry, shared subscriptions, topic aliases, user properties and a request-response pattern. A broker that supports 5.0 also accepts 3.1.1 clients.

"Lightweight" is meant literally. The fixed header of an MQTT packet is two bytes for a small message; the keep-alive ping (PINGREQ) is exactly two bytes. The length field uses one to four bytes, seven bits per byte, with the top bit meaning "another byte follows". This example builds the packet that a P1 dongle sends for one power reading:

def remaining_length(n):
    """MQTT variable byte integer: 7 bits per byte, high bit = more follows."""
    out = bytearray()
    while True:
        byte, n = n % 128, n // 128
        out.append(byte | 0x80 if n else byte)
        if not n:
            return bytes(out)


def publish_qos1(topic, payload, packet_id):
    t = topic.encode()
    body = len(t).to_bytes(2, "big") + t + packet_id.to_bytes(2, "big") + payload
    return bytes([0x32]) + remaining_length(len(body)) + body   # 0x32 = PUBLISH, QoS 1


for n in (127, 128, 321, 16384):
    print(f"{n:6} -> {remaining_length(n).hex(' ').upper()}")

payload = b'{"time":"2026-10-07T19:30:15+02:00","watts":1234}'
packet = publish_qos1("kestrel/home/h0017/p1/power", payload, 1)
print(f"payload {len(payload)} bytes, whole PUBLISH packet {len(packet)} bytes")
print("first bytes:", packet[:4].hex(" ").upper())
print("PINGREQ:", bytes([0xC0, 0x00]).hex(" ").upper())

   127 -> 7F
   128 -> 80 01
   321 -> C1 02
 16384 -> 80 80 01
payload 49 bytes, whole PUBLISH packet 82 bytes
first bytes: 32 50 00 1B
PINGREQ: C0 00

A 49-byte reading costs 82 bytes on the wire: one byte for the packet type and flags (0x32), one for the length (0x50 = 80), two for the topic length (0x001B = 27), the topic itself, a two-byte packet identifier and the payload. The topic is often the biggest part of the overhead, which is why MQTT 5.0 added topic aliases. TLS adds its own overhead on top, but once per connection, not per message.

The broker

An MQTT broker is the server every MQTT client connects to: it accepts published messages, checks whether the client is allowed to publish them, and forwards a copy to every client with a matching subscription. It also keeps retained messages and sessions, and publishes a client's last will when that client disappears. It is a message broker specialized for many small, unreliable clients. Eclipse Mosquitto is a common open-source broker (2.1.2 is the current upstream release as of October 2026; Debian 12 ships 2.0.11, which recorded the output on this page); EMQX, HiveMQ and VerneMQ are others, and the cloud IoT services run their own (see IoT Platforms and the Cloud).

This topic has two brokers in its story. Lotte Kortenhoeve runs Mosquitto next to Home Assistant on her Raspberry Pi 5 at 192.168.1.20; her own ESP32 reader publishes live power to home/p1/power, and nothing leaves her home. The Kestrel Home pilot, which started in September 2026, uses Kestrel's own broker: about 250 P1 dongles publish to kestrel/home/<home-id>/p1/power over TLS, with a certificate per dongle. ADR-0011, still proposed, records that design. Lotte's dongle is home h0017; the output below uses the pilot's topic layout.

Publishing and subscribing

The Mosquitto package for Debian and Ubuntu comes in two parts: mosquitto (the broker) and mosquitto-clients (the command-line tools mosquitto_pub and mosquitto_sub). The clients are the quickest way to see what a broker does, and the first tools to reach for when a device "sends but nothing arrives".

A dashboard subscribes first and keeps running. -v prints the topic before each payload, -q 1 asks for quality of service 1, and -i sets the client identifier, which must be unique on the broker:

mosquitto_sub -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-1 -v -q 1 -t 'kestrel/home/+/p1/power'

Then the dongle of home h0017 publishes one reading:

mosquitto_pub -h 127.0.0.1 -p 1883 -u p1-h0017 -P "$MQTT_PASSWORD" -i p1-h0017 -q 1 -t kestrel/home/h0017/p1/power -m '{"time":"2026-10-07T19:30:15+02:00","watts":1234}'

The publisher prints nothing and exits with status 0. The subscriber prints the topic and the payload:

kestrel/home/h0017/p1/power {"time":"2026-10-07T19:30:15+02:00","watts":1234}

Three more readings later, the dashboard shows:

kestrel/home/h0017/p1/power {"time":"2026-10-07T19:30:15+02:00","watts":1234}
kestrel/home/h0017/p1/power {"time":"2026-10-07T19:30:25+02:00","watts":1187}
kestrel/home/h0017/p1/power {"time":"2026-10-07T19:30:35+02:00","watts":1190}
kestrel/home/h0017/p1/power {"time":"2026-10-07T19:30:45+02:00","watts":1201}

The payload is just bytes to the broker: MQTT does not know or care that it is JSON. The format is an agreement between publishers and subscribers, so write it down. The pilot puts the time of the measurement and the unit in every message, as the protocol translation rules on the gateways page recommend; a dashboard that only sees 1234 cannot tell watts from kilowatts or know when the value was measured.

Topics and wildcards

A topic is the name a message is published under: a UTF-8 string of levels separated by forward slashes, such as kestrel/home/h0017/p1/power, which the broker uses to decide which subscribers receive it. Topics do not need to be created first; publishing to a new name is enough. Three rules cause most surprises:

• Topics are case-sensitive. Kestrel/home/... and kestrel/home/... are different topics.
• A leading slash adds a level. /kestrel/home starts with an empty level, so it is a different topic from kestrel/home, with one more level.
• Topics starting with $ are reserved for the broker. Mosquitto publishes its statistics under $SYS/; that prefix is a widely used convention, not part of the standard.

Wildcards

A topic wildcard is a special level in a subscription filter that matches many topics at once: + matches exactly one level, and #, which may only be the last level, matches that level and every level below it. Wildcards are only for subscribing; a publish always goes to one concrete topic. With three test messages published (power for h0017 and h0018, plus a made-up gas reading of 2841.517 m³ for h0017 to show a second topic; the pilot itself sends only power), each subscription filter received:

The recorded output for kestrel/home/h0017/#:

kestrel/home/h0017/p1/power {"time":"2026-10-07T19:31:15+02:00","watts":1234}
kestrel/home/h0017/p1/gas {"time":"2026-10-07T19:30:00+02:00","m3":2841.517}

A filter with a wildcard in the wrong place is rejected by the client before it even connects:

mosquitto_sub -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-1 -t 'kestrel/#/power'

Error: Invalid subscription topic 'kestrel/#/power', are all '+' and '#' wildcards correct?

Use 'mosquitto_sub --help' to see usage.

The matching rules are short enough to write down in a dozen lines, which is useful for a gateway or a test that needs to know which messages a subscription will see:

def matches(topic_filter, topic):
    """MQTT topic matching: '+' is exactly one level, '#' this level and below."""
    if topic.startswith("$") and topic_filter[:1] in ("+", "#"):
        return False                  # wildcards at the start never match $SYS
    f_levels = topic_filter.split("/")
    t_levels = topic.split("/")
    for i, f in enumerate(f_levels):
        if f == "#":
            return True               # also matches the parent level itself
        if i >= len(t_levels):
            return False
        if f != "+" and f != t_levels[i]:
            return False
    return len(f_levels) == len(t_levels)


tests = [
    ("kestrel/home/+/p1/power", "kestrel/home/h0017/p1/power"),
    ("kestrel/home/+/p1/power", "kestrel/home/h0017/p1/gas"),
    ("kestrel/home/h0017/#", "kestrel/home/h0017"),
    ("kestrel/home/h0017/#", "kestrel/home/h0017/p1/power"),
    ("kestrel/home/h0017/+", "kestrel/home/h0017/p1/power"),
    ("sport/+", "sport/"),
    ("#", "$SYS/broker/version"),
    ("$SYS/#", "$SYS/broker/version"),
    ("kestrel/home/h0017/p1/power", "Kestrel/home/h0017/p1/power"),
    ("kestrel/home/h0017/p1/power", "/kestrel/home/h0017/p1/power"),
]
for f, t in tests:
    print(f"{f:28} {t:29} {matches(f, t)}")

kestrel/home/+/p1/power      kestrel/home/h0017/p1/power   True
kestrel/home/+/p1/power      kestrel/home/h0017/p1/gas     False
kestrel/home/h0017/#         kestrel/home/h0017            True
kestrel/home/h0017/#         kestrel/home/h0017/p1/power   True
kestrel/home/h0017/+         kestrel/home/h0017/p1/power   False
sport/+                      sport/                        True
#                            $SYS/broker/version           False
$SYS/#                       $SYS/broker/version           True
kestrel/home/h0017/p1/power  Kestrel/home/h0017/p1/power   False
kestrel/home/h0017/p1/power  /kestrel/home/h0017/p1/power  False

Two lines deserve a second look. kestrel/home/h0017/# also matches kestrel/home/h0017 itself: # includes the parent level. And sport/+ matches sport/, because the empty string after the slash is a level too; Mosquitto 2.0.11 behaved the same way when we tried it, as the MQTT specification requires.

The $SYS topics

Mosquitto publishes its own version, client counts, message rates and more under $SYS/, updated every sys_interval seconds. The dashboard user may read them:

mosquitto_sub -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-1 -t '$SYS/broker/version' -C 1

mosquitto version 2.0.11

Subscribing to # does not include them, even for a user who has read access to $SYS/#. In 12 seconds, while the broker updated its statistics and one power reading came in, this subscription received only the normal topics:

mosquitto_sub -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-1 -v -t '#' -W 12

kestrel/home/h0017/status online
kestrel/home/h0017/p1/power {"time":"2026-10-07T19:31:05+02:00","watts":1228}
Timed out

-W 12 stops after 12 seconds and prints Timed out (exit status 27). The status line is a retained message, explained below.

Designing a topic tree

Topic names are an interface: once 250 dongles publish to them, changing them means a firmware update. A few rules keep a tree usable for years:

• From general to specific. kestrel/home/<home-id>/p1/power lets a subscriber pick one home (kestrel/home/h0017/#) or one kind of value for all homes (kestrel/home/+/p1/power).
• Make access control follow the tree. When every device's topics sit under its own level, one access rule per device is enough: p1-h0017 may write kestrel/home/h0017/# and nothing else.
• No personal identifiers in topics. Topic names end up in broker logs, dashboards and debugging sessions. The pilot uses a pilot home number, never the EAN code or the customer number, for the same reason metrics never carry the EAN as a label.
• Lowercase, no leading slash, no spaces. It avoids the case and empty-level surprises above.

Quality of service

Quality of service (QoS) in MQTT is the delivery guarantee a client asks for on one connection: 0, 1 or 2, each with its own acknowledgment exchange. The levels map onto the general delivery guarantees:

QoS on the wire

With -d the clients print every packet they send and receive. A QoS 1 publish is acknowledged with one PUBACK:

mosquitto_pub -d -h 127.0.0.1 -p 1883 -u p1-h0017 -P "$MQTT_PASSWORD" -i p1-h0017 -q 1 -t kestrel/home/h0017/p1/power -m '{"time":"2026-10-07T19:30:25+02:00","watts":1187}'

Client p1-h0017 sending CONNECT
Client p1-h0017 received CONNACK (0)
Client p1-h0017 sending PUBLISH (d0, q1, r0, m1, 'kestrel/home/h0017/p1/power', ... (49 bytes))
Client p1-h0017 received PUBACK (Mid: 1, RC:0)
Client p1-h0017 sending DISCONNECT

In the PUBLISH line, d is the duplicate flag (set on a resend), q the QoS, r the retain flag and m the packet identifier. The same command with -q 2 needs the four-step exchange:

Client p1-h0017 sending CONNECT
Client p1-h0017 received CONNACK (0)
Client p1-h0017 sending PUBLISH (d0, q2, r0, m1, 'kestrel/home/h0017/p1/power', ... (49 bytes))
Client p1-h0017 received PUBREC (Mid: 1)
Client p1-h0017 sending PUBREL (m1)
Client p1-h0017 received PUBCOMP (Mid: 1, RC:0)
Client p1-h0017 sending DISCONNECT

QoS applies per hop

A PUBACK means "the broker has it", not "the subscriber has it". The broker log of the first QoS 1 publish shows the order: the broker forwards the message to the two connected dashboards, then acknowledges the publisher, and only after that do the dashboards acknowledge their copies:

1791394215: Received PUBLISH from p1-h0017 (d0, q1, r0, m1, 'kestrel/home/h0017/p1/power', ... (49 bytes))
1791394215: Sending PUBLISH to dash-1 (d0, q1, r0, m1, 'kestrel/home/h0017/p1/power', ... (49 bytes))
1791394215: Sending PUBLISH to dash-2 (d0, q1, r0, m1, 'kestrel/home/h0017/p1/power', ... (49 bytes))
1791394215: Sending PUBACK to p1-h0017 (m1, rc0)
1791394215: Received PUBACK from dash-1 (Mid: 1, RC:0)
1791394215: Received PUBACK from dash-2 (Mid: 1, RC:0)

The QoS of the second hop is not the publisher's choice either. A message is delivered with the lower of two values: the QoS it was published with and the maximum QoS of the subscription. This dashboard subscribed with QoS 1 and received all four readings published above; the one sent with QoS 2 arrived as q1, and the one sent with QoS 0 arrived as q0, without a PUBACK:

mosquitto_sub -d -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-2 -q 1 -t 'kestrel/home/+/p1/power'

Client dash-2 sending CONNECT
Client dash-2 received CONNACK (0)
Client dash-2 sending SUBSCRIBE (Mid: 1, Topic: kestrel/home/+/p1/power, QoS: 1, Options: 0x00)
Client dash-2 received SUBACK
Subscribed (mid: 1): 1
Client dash-2 received PUBLISH (d0, q1, r0, m1, 'kestrel/home/h0017/p1/power', ... (49 bytes))
Client dash-2 sending PUBACK (m1, rc0)
{"time":"2026-10-07T19:30:15+02:00","watts":1234}
Client dash-2 received PUBLISH (d0, q1, r0, m2, 'kestrel/home/h0017/p1/power', ... (49 bytes))
Client dash-2 sending PUBACK (m2, rc0)
{"time":"2026-10-07T19:30:25+02:00","watts":1187}
Client dash-2 received PUBLISH (d0, q1, r0, m3, 'kestrel/home/h0017/p1/power', ... (49 bytes))
Client dash-2 sending PUBACK (m3, rc0)
{"time":"2026-10-07T19:30:35+02:00","watts":1190}
Client dash-2 received PUBLISH (d0, q0, r0, m0, 'kestrel/home/h0017/p1/power', ... (49 bytes))
{"time":"2026-10-07T19:30:45+02:00","watts":1201}
Client dash-2 sending DISCONNECT

Subscribed (mid: 1): 1 is the QoS the broker granted. The packet identifiers on this hop (m1, m2, m3) are the broker's own numbering, not the publisher's.

Sessions and keep alive

Two connection settings belong with QoS. The keep alive is the longest time a client promises to stay silent; if nothing else needs sending, it sends a two-byte PINGREQ. The broker treats a client as gone after one and a half times the keep alive without any traffic: 90 seconds for the 60-second default of the Mosquitto clients. The session decides what survives a disconnect. With a clean session (MQTT 3.1.1) or Clean Start plus a session expiry of zero (MQTT 5.0), the broker forgets the client when it leaves. With a persistent session, the broker keeps its subscriptions and queues QoS 1 and 2 messages while it is offline, which suits a backend consumer that must not miss readings, but costs broker memory for every absent client.

Retained messages and last will

A dashboard that starts at 19:31 has a problem: it knows nothing until each device publishes again. And when a dongle loses power, it cannot say goodbye. MQTT has one feature for each.

Retained messages

A retained message is a message published with the retain flag set, which the broker stores as the last known value of its topic and sends immediately to every new subscriber of that topic. A topic holds at most one retained message; a newer one replaces it. The dongle publishes its status retained:

mosquitto_pub -h 127.0.0.1 -p 1883 -u p1-h0017 -P "$MQTT_PASSWORD" -i p1-h0017 -q 1 -r -t kestrel/home/h0017/status -m online

A dashboard that subscribes afterward gets it at once (-C 1 stops after one message):

mosquitto_sub -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-1 -v -t 'kestrel/home/+/status' -C 1

kestrel/home/h0017/status online

With -d the delivery shows r1: the subscriber can tell a stored value from a fresh one. mosquitto_sub -R hides messages with that flag:

mosquitto_sub -h 127.0.0.1 -p 1883 -u dashboard -P "$MQTT_PASSWORD" -i dash-1 -v -R -t 'kestrel/home/+/status' -W 3

Timed out

A retained message stays until it is replaced or deleted, also when the device is long gone. To delete it, publish an empty retained message (-n sends a zero-length payload):

mosquitto_pub -d -h 127.0.0.1 -p 1883 -u p1-h0017 -P "$MQTT_PASSWORD" -i p1-h0017 -q 1 -r -n -t kestrel/home/h0017/status

Client p1-h0017 sending CONNECT
Client p1-h0017 received CONNACK (0)
Client p1-h0017 sending PUBLISH (d0, q1, r1, m1, 'kestrel/home/h0017/status', ... (0 bytes))
Client p1-h0017 received PUBACK (Mid: 1, RC:0)
Client p1-h0017 sending DISCONNECT

Retain status values and settings, not readings: a retained power value from last week looks exactly like a current one to a new subscriber unless the payload carries its measurement time.

Last will

A last will and testament (or will message) is a message a client hands to the broker when it connects, which the broker publishes on the client's behalf if the connection ends without a proper DISCONNECT: a power cut, a network failure or a keep-alive timeout. Combined with a retained status, it gives every dashboard an accurate online or offline flag. In the lab, a stand-in for the dongle connected with a will:

mosquitto_sub -h 127.0.0.1 -p 1883 -u p1-h0017 -P "$MQTT_PASSWORD" -i p1-h0017 --will-topic kestrel/home/h0017/status --will-payload offline --will-retain --will-qos 1 -t kestrel/home/h0017/cmd

It then announced itself online (retained), and the process was killed, the software equivalent of pulling the plug. A dashboard watching kestrel/home/+/status printed:

kestrel/home/h0017/status online
kestrel/home/h0017/status offline

Because the will was retained, a dashboard that starts later still sees offline. The broker log tells the two endings apart. After the kill:

1791394267: Client p1-h0017 closed its connection.
1791394267: Sending PUBLISH to dash-1 (d0, q0, r0, m0, 'kestrel/home/h0017/status', ... (7 bytes))

In a second round the will client was stopped with Ctrl+C, which makes it send DISCONNECT. The dashboard printed only online, and the log said:

1791394280: Received DISCONNECT from p1-h0017
1791394280: Client p1-h0017 disconnected.

In the lab the will followed within a second, because the operating system closed the dead process's socket. A dongle whose Wi-Fi drops closes nothing: the broker only notices when the keep alive runs out, after 90 seconds with a 60-second keep alive. Choose the keep alive with that delay in mind.

Securing MQTT

MQTT itself encrypts nothing. On port 1883, user names, passwords and every reading cross the network in plain text, and a broker that accepts anonymous clients lets anyone read and write every topic. Search engines for internet-connected devices regularly find such brokers. Mosquitto 2.0 made the safe choice the default: without a configured listener it only listens on the loopback address, and a configured listener refuses anonymous clients unless allow_anonymous true is set.

Authentication

The lab broker used this configuration:

listener 1883 127.0.0.1
allow_anonymous false
password_file /etc/mosquitto/passwd
acl_file /etc/mosquitto/acl
persistence false
sys_interval 10
log_dest file /var/log/mosquitto/mosquitto.log
log_type all

log_type all is only there so the log shows every packet; leave it out in production. The password file holds salted hashes and is managed with mosquitto_passwd, which prompts for the password when you leave it off the command line. -c creates a new file and overwrites an existing one, so use it only for the first user:

mosquitto_passwd -c /etc/mosquitto/passwd dashboard
mosquitto_passwd /etc/mosquitto/passwd p1-h0017

A client without credentials is refused:

mosquitto_sub -h 127.0.0.1 -p 1883 -t '#'

Connection error: Connection Refused: not authorised.

A wrong password gives exactly the same message and exit status 5, and the broker log does not distinguish the two cases either. That is deliberate: an attacker cannot use the error to find out which user names exist. With -V mqttv5 the wording changes to Connection error: Not authorized and the exit status is 135, the MQTT 5.0 reason code itself.

Access control per topic

Authentication says who a client is; the access control list (ACL) says which topics it may read and write. The lab's ACL file:

# Dashboard: read-only on all homes, plus broker statistics
user dashboard
topic read kestrel/home/#
topic read $SYS/#

# P1 dongle of home h0017: may only write its own topics
user p1-h0017
topic write kestrel/home/h0017/#

# P1 dongle of home h0018
user p1-h0018
topic write kestrel/home/h0018/#

What happens when the h0017 dongle publishes to h0018's topic depends on the protocol version. With MQTT 3.1.1, the default of the clients, the broker acknowledges the message and drops it. The publisher cannot tell:

mosquitto_pub -d -h 127.0.0.1 -p 1883 -u p1-h0017 -P "$MQTT_PASSWORD" -i p1-h0017 -q 1 -t kestrel/home/h0018/p1/power -m '{"time":"2026-10-07T19:32:00+02:00","watts":99999}'

Client p1-h0017 sending CONNECT
Client p1-h0017 received CONNACK (0)
Client p1-h0017 sending PUBLISH (d0, q1, r0, m1, 'kestrel/home/h0018/p1/power', ... (50 bytes))
Client p1-h0017 received PUBACK (Mid: 1, RC:0)
Client p1-h0017 sending DISCONNECT

With MQTT 5.0 (-V mqttv5), the PUBACK carries reason code 135, "Not authorized":

Client p1-h0017 sending CONNECT
Client p1-h0017 received CONNACK (0)
Client p1-h0017 sending PUBLISH (d0, q1, r0, m1, 'kestrel/home/h0018/p1/power', ... (50 bytes))
Client p1-h0017 received PUBACK (Mid: 1, RC:135)
Warning: Publish 1 failed: Not authorized.
Client p1-h0017 sending DISCONNECT

In both cases the dashboard received nothing, and mosquitto_pub still exited with status 0, so a script cannot rely on the exit status to detect the refusal. With QoS 0 there is no PUBACK, and even MQTT 5.0 gives no warning.

Reading is checked differently. A client that subscribes to a topic it may not read still gets a normal SUBACK (granted QoS 0 in the lab); Mosquitto checks read access when it delivers a message, so the messages simply never arrive. A subscription that "works" but stays silent can therefore be an ACL problem.

What the broker log shows

Without a log_type line, Mosquitto logs connections and disconnections, but not individual packets. This run contained an anonymous attempt, an allowed publish, a refused publish with MQTT 5.0, a client that was killed and one that was stopped normally:

1791394429: mosquitto version 2.0.11 starting
1791394429: Config loaded from /etc/mosquitto/mosquitto.conf.
1791394429: Opening ipv4 listen socket on port 1883.
1791394429: mosquitto version 2.0.11 running
1791394430: New connection from 127.0.0.1:42250 on port 1883.
1791394430: Client <unknown> disconnected, not authorised.
1791394430: New connection from 127.0.0.1:42260 on port 1883.
1791394430: New client connected from 127.0.0.1:42260 as p1-h0017 (p2, c1, k60, u'p1-h0017').
1791394430: Client p1-h0017 disconnected.
1791394430: New connection from 127.0.0.1:42264 on port 1883.
1791394430: New client connected from 127.0.0.1:42264 as p1-h0017 (p5, c1, k60, u'p1-h0017').
1791394430: Client p1-h0017 disconnected.
1791394430: New connection from 127.0.0.1:42280 on port 1883.
1791394430: New client connected from 127.0.0.1:42280 as p1-h0017 (p2, c1, k60, u'p1-h0017').
1791394431: Client p1-h0017 closed its connection.
1791394432: New connection from 127.0.0.1:42286 on port 1883.
1791394432: New client connected from 127.0.0.1:42286 as p1-h0017 (p2, c1, k60, u'p1-h0017').
1791394433: Client p1-h0017 disconnected.
1791394434: mosquitto version 2.0.11 terminating

The timestamps are Unix seconds. In (p2, c1, k60, u'p1-h0017'), p2 is MQTT 3.1.1 and p5 MQTT 5.0, c1 a clean session, k60 a 60-second keep alive and u the user name. The refused publish on the third connection looks exactly like a successful one: the Denied PUBLISH line only appears at debug level (log_type all or log_type debug). closed its connection marks an abrupt end, disconnected a clean one.

TLS in production

Outside a lab, use TLS on port 8883, also on a home network if the broker is reachable from other devices. The pilot goes one step further: each dongle has its own key pair and a certificate issued by Kestrel's device CA, and the broker accepts only clients that present such a certificate (mutual TLS). How those certificates get onto the devices is on Provisioning and Over-the-Air Updates. In Mosquitto it looks roughly like this:

listener 8883
cafile /etc/mosquitto/ca_certificates/kestrel-home-device-ca-1.pem
certfile /etc/mosquitto/certs/broker.pem
keyfile /etc/mosquitto/certs/broker.key
require_certificate true
crlfile /etc/mosquitto/ca_certificates/kestrel-home-device-ca-1.crl
use_identity_as_username true
allow_anonymous false
acl_file /etc/mosquitto/acl

use_identity_as_username makes the certificate's Common Name (for Lotte's dongle, KHP1-000231) the user name that the ACL checks, so the rule pairing that dongle with home h0017 grants it topic write kestrel/home/h0017/#. A stolen dongle is then one revoked certificate (the broker checks the CA's revocation list, crlfile), not a shared password to change on 250 devices. The device-side requirements (no default passwords, a support period, signed updates) are on IoT Security.

Reconnecting without a storm

Devices lose their broker all the time: a Wi-Fi hiccup, a router restart, a broker upgrade. The reconnect logic in the firmware decides what happens next, and with a fleet, that logic runs on every device at the same moment.

On the evening of Monday 28 September 2026, Kestrel restarted the pilot broker in a planned change. About 250 dongles, all on firmware 1.3.2, lost their connection at the same instant and tried to reconnect at the same instant. Each new connection costs the broker a TLS handshake with a certificate check, and 250 at once were more than it could handle; most attempts timed out. Firmware 1.3.2 retried after a fixed five seconds, without any randomness, so the dongles that failed together tried again together, and collided again. For a while the app showed no live data at all for the pilot homes, long after the broker itself was back. This became incident INC-33004.

The fix is the standard one from backoff and jitter: exponential backoff makes each device wait longer after every failure, and a random delay ("full jitter") breaks up the groups. This simulation compares both with a simplified broker that can set up 50 connections per second and wastes its time on handshakes that time out when more arrive:

import random
from collections import Counter

DONGLES = 250
CAPACITY = 50           # new connections per second the broker can handle
RETRY_S = 5             # firmware 1.3.2: fixed retry interval, no jitter
BASE_S, CAP_S = 5, 120  # firmware 1.4.0: exponential backoff, full jitter


def accepted(attempts):
    """Simplified broker: when overloaded, handshakes time out and waste work."""
    if attempts <= CAPACITY:
        return attempts
    return CAPACITY * CAPACITY // attempts


def fixed(rng, failures):
    return 0 if failures == 0 else RETRY_S


def backoff_with_jitter(rng, failures):
    return rng.uniform(0, min(CAP_S, BASE_S * 2 ** failures))


def simulate(delay):
    rng = random.Random(33004)          # fixed seed: the same run every time
    failures = Counter()
    due = {d: delay(rng, 0) for d in range(DONGLES)}   # broker back at t = 0
    per_second = Counter()
    second = 0
    while due:
        now = sorted(d for d, t in due.items() if int(t) == second)
        per_second[second] = len(now)
        rng.shuffle(now)
        ok = accepted(len(now))
        for d in now[:ok]:
            del due[d]                  # connected
        for d in now[ok:]:
            failures[d] += 1
            due[d] = second + max(1, delay(rng, failures[d]))
        second += 1
    return per_second, second


for name, strategy in [("1.3.2 fixed 5 s", fixed),
                       ("1.4.0 backoff+jitter", backoff_with_jitter)]:
    per_second, done = simulate(strategy)
    print(f"{name:21} peak {max(per_second.values()):3} attempts/s, "
          f"{sum(per_second.values()):4} attempts, all connected after {done:2} s")
    print(f"{'':21} first 12 s: {[per_second[s] for s in range(12)]}")

1.3.2 fixed 5 s       peak 250 attempts/s, 2397 attempts, all connected after 71 s
                      first 12 s: [250, 0, 0, 0, 0, 240, 0, 0, 0, 0, 230, 0]
1.4.0 backoff+jitter  peak  55 attempts/s,  278 attempts, all connected after 14 s
                      first 12 s: [54, 55, 53, 48, 52, 1, 2, 4, 1, 3, 2, 1]

With a fixed interval, the whole fleet arrives as one wave every five seconds, and each wave is too big to get through, so the broker needs more than a minute and almost 2,400 attempts. With a random first delay of up to five seconds, the same 250 dongles arrive at about the rate the broker can take: 278 attempts, all connected in 14 seconds. The numbers come from a deliberately simple model; the shape, one huge spike that keeps coming back versus a spread-out trickle, is what real reconnect storms look like.

Firmware 1.4.0 adds exponential backoff with random jitter to every reconnect, including the first one. It is rolled out under change CHG-5012 in three waves, starting Thursday 1 October 2026 (see staged rollouts). Until every dongle runs 1.4.0, planned broker restarts for the pilot happen at night, when few people watch live data.

Alternatives: CoAP and HTTP

MQTT is not the only way for a device to send data, and not always the best one.

CoAP (Constrained Application Protocol, RFC 7252, 2014) is a web-style request-response protocol for very small devices and lossy networks: it runs over UDP (port 5683, or 5684 with DTLS, the datagram version of TLS), has a fixed header of only four bytes, and uses the methods GET, POST, PUT and DELETE on resources with URIs, much like HTTP. Because UDP does not guarantee delivery, CoAP has its own: a confirmable (CON) message is resent until the other side replies with an acknowledgment (ACK); a non-confirmable (NON) message is sent once; a reset (RST) rejects a message. An extension, Observe (RFC 7641), lets a client subscribe to a resource and receive changes. CoAP is used in low-power wireless mesh networks, and the device-management standard LwM2M is built on it.

Plain HTTPS is often the simplest choice for a device that reports rarely: a weather station that uploads once an hour, or a device that sends a daily summary. Every web server, proxy and load balancer understands it. The costs are headers on every request, and a TLS handshake per connection if the device does not keep the connection open, which matters on a battery. Browsers cannot open raw TCP connections, so a browser dashboard that wants live MQTT data uses MQTT over WebSockets, which most brokers, Mosquitto included, can offer on a separate listener.

Two relatives of MQTT are worth recognizing by name: MQTT-SN, a variant for sensor networks without TCP/IP, and Sparkplug, a specification on top of MQTT that fixes topic names and payloads for industrial equipment (see the Industrial Internet of Things).

Troubleshooting MQTT

Most MQTT problems come down to a handful of causes. Start with mosquitto_sub -v on the broker side and -d on the client side, so you can see what really travels.