Edge Computing and Gateways

Why process data at the edge

Edge computing means processing data close to where it is produced instead of sending every raw value to a central data center or the cloud. The OT topic gives the four classic reasons: latency, autonomy, bandwidth and control of the boundary. In IoT two more are worth naming:

• Cost per message. Cloud IoT services, mobile data and battery power are all paid per message or per byte. A device that sends ten times less costs roughly ten times less to run.
• Privacy. Data that never leaves the home or the site cannot leak from a cloud service. Aggregating before sending is also a form of data minimization.

"Edge" is a relative word. For Lotte Kortenhoeve, the Kestrel customer whose home runs through this topic, the edge is her hallway: an ESP32 board on the smart meter's P1 port receives a telegram every second and passes it over Wi-Fi to a Raspberry Pi 5 running the Mosquitto MQTT broker and Home Assistant. The Pi keeps the one-second data for a day, stores 15-minute aggregates, and controls her home battery. Nothing leaves the house. Two other meanings of "edge" on this site are different things: a cloud provider's edge location brings content closer to users, and an edge data center is a small data center near its users. Both are still far from a meter cupboard.

What a gateway does

An IoT gateway is a device that connects local devices to a wider network or platform and does work on the data on the way. Many devices cannot reach the internet on their own: a Zigbee or Bluetooth Low Energy sensor has no IP address, a meter has a serial port, a factory controller speaks Modbus. The gateway is the one box that can talk to both sides.

A gateway usually does some or all of these jobs:

• Protocol translation: from the devices' protocols to the one the platform expects (next section).
• Filtering and aggregation: sending less, and sending it in a better shape.
• Buffering: keeping data during an outage and sending it later.
• Local decisions: rules that must work without the cloud.
• A security boundary: devices without an IP address never reach the internet; the gateway holds the credentials and the one outgoing connection, and can be hardened and monitored in one place.

Kinds of gateways

Two of these are special cases. A LoRaWAN gateway forwards every packet it hears to a network server and cannot read the application data, which is encrypted between the device and the application (see Wireless Basics). And a smart meter's telecom module, which Netbeheer Nederland calls the meter's "gateway", belongs to the grid operator and only serves the grid operator's own data path. On these IoT pages, a gateway is always on the customer's or the site's side.

Protocol translation

Protocol translation is converting data from the protocol and format a device speaks into the protocol and format another system expects, without changing its meaning. Lotte's reader does it on a small scale: it receives P1 telegrams over a serial line, checks their CRC, and publishes the values it needs as MQTT messages to the broker on her Pi. One telegram becomes a handful of messages (illustrative topics; the MQTT side is explained on MQTT and IoT Messaging):

0-0:1.0.0(261007193015S)
1-0:1.8.2(000467.905*kWh)
1-0:1.7.0(01.234*kW)
1-0:2.7.0(00.000*kW)

home/p1/energy_import_t2  {"time":"2026-10-07T19:30:15+02:00","kwh":467.905}
home/p1/power             {"time":"2026-10-07T19:30:15+02:00","watts":1234}

Translation is where meaning gets lost if you are careless. Things to carry across explicitly:

• Units and scale. The meter says kW with three decimals; the message says watts as an integer. Put the unit in the field name or the payload, never only in someone's head.
• Time. Use the time the value was measured, with its offset (the telegram's S or W flag becomes +02:00), not the time the gateway forwarded it.
• Quality. A value from a telegram with a bad CRC should not become a normal-looking message. Drop it, or mark it, and count the failures.
• Identity. Decide which identifier goes into the topic. Lotte's own topics say home/p1; the pilot uses a pilot home number, never a meter number or EAN code.

Ready-made translators exist for most common cases: Home Assistant and ESPHome have a DSMR integration, open-source bridges such as Zigbee2MQTT turn Zigbee devices into MQTT topics, and industrial gateways map Modbus registers (see Modbus) to MQTT or OPC UA. Writing your own is mainly worth it when nothing fits, as with the pilot dongle's firmware.

Reducing data at the edge

Edge filtering is deciding at the edge which data is worth sending: dropping invalid values, aggregating many values into a few, and sending only what changed. The question to ask is what the receiver actually needs. Home Assistant's energy dashboard needs energy per hour; a live power display needs a value every few seconds; nobody needs 86,400 power values a day in the cloud.

Aggregating: a quarter hour in one record

Lotte's Pi turns 900 one-second values into one record per quarter hour, with the mean, minimum and maximum power. For the energy it does not add up the power values but subtracts two meter register readings, and the example shows why. In this quarter hour the oven and then a kettle were on, and a Wi-Fi hiccup lost 20 telegrams:

from statistics import mean

# One quarter hour, 19:30:00-19:44:59, one P1 telegram per second.
def power_at(second):            # watts imported, invented but realistic
    if second < 300:
        return 1234              # oven still on
    if second < 480:
        return 3434              # plus a 2,200 W kettle
    return 610                   # evening base load

LOST = range(400, 420)           # Wi-Fi hiccup: 20 telegrams never arrive
received = [power_at(s) for s in range(900) if s not in LOST]

REGISTER_START = 467.900         # 1-0:1.8.2 at 19:30:00, kWh
REGISTER_END = 468.245           # 1-0:1.8.2 at 19:45:00, kWh

integrated_wh = sum(received) / 3600          # each sample stands for 1 s
register_wh = (REGISTER_END - REGISTER_START) * 1000

print(f"samples     {len(received)} of 900")
print(f"mean power  {mean(received):.0f} W")
print(f"min / max   {min(received)} / {max(received)} W")
print(f"energy from summing power     {integrated_wh:6.1f} Wh")
print(f"energy from the register      {register_wh:6.1f} Wh")

samples     880 of 900
mean power  1336 W
min / max   610 / 3434 W
energy from summing power      326.6 Wh
energy from the register       345.0 Wh

Summing the power values ("integrating") misses the 20 lost seconds of oven and kettle, so it reports about 18 Wh less than the register. Missing data also pulls the mean down: the true mean over all 900 seconds is 1383 W. The register difference does not care how many telegrams arrived: the meter counted every watt-hour itself, and the only error is its 1 Wh resolution. Rule: take energy from the registers and power statistics from the samples, and record how many samples the statistics are based on. More on windows, gaps and late data is on IoT Data: Time Series and Streams.

Deadbands: report by exception

A deadband is a range of change that is ignored: a new value is only reported when it differs from the last reported value by at least a set amount. It is the IoT form of the report by exception known from industrial protocols. A deadband on its own has a weakness: when nothing changes, nothing is sent, and the receiver cannot tell a quiet house from a dead device. So it is combined with a heartbeat, a message at least every so often.

import random

random.seed(17)                  # invented but repeatable data

# Ten minutes of one-second power values: steady use with a little noise,
# and a washing machine heating up from second 240 to second 420.
readings = []
for second in range(600):
    heater = 1900 if 240 <= second < 420 else 0
    readings.append(520 + heater + random.randint(-20, 20))

DEADBAND_W = 50                  # report a change of 50 W or more...
HEARTBEAT_S = 60                 # ...and at least once a minute anyway

sent = []
for second, watts in enumerate(readings):
    if (not sent
            or abs(watts - sent[-1][1]) >= DEADBAND_W
            or second - sent[-1][0] >= HEARTBEAT_S):
        sent.append((second, watts))

print(f"readings: {len(readings)}, messages sent: {len(sent)}")
for second, watts in sent:
    print(f"  t={second:3} s  {watts:4} W")

readings: 600, messages sent: 10
  t=  0 s   533 W
  t= 60 s   533 W
  t=120 s   505 W
  t=180 s   506 W
  t=240 s  2421 W
  t=300 s  2418 W
  t=360 s  2425 W
  t=420 s   528 W
  t=480 s   506 W
  t=540 s   520 W

Ten messages instead of 600, and the washing machine's start and stop are still there to the second. Seven of the ten are heartbeats. The size of the deadband is a trade-off: too small and the noise gets through, too large and real changes are lost. Pick it from what the receiver needs, not from what the device can do.

The Kestrel Home pilot uses the simplest reduction: each dongle sends one 10-second average of the power instead of every one-second telegram. For 250 dongles that is 250 × 8,640 = 2.16 million messages a day instead of 250 × 86,400 = 21.6 million, ten times less traffic, storage and cost for a live display that looks the same to a person.

Store and forward

Store and forward means keeping messages in a local buffer while the connection or the receiver is unavailable, and sending them once it is back. Without it, every broker restart, Wi-Fi drop or mobile dead spot becomes a gap in the data. With it, three questions need an answer:

• How big is the buffer? Memory and flash are limited, so the buffer must be bounded. An ESP32 can keep minutes or hours in RAM; a Pi with an SSD can keep weeks.
• What is dropped when it is full? Usually the oldest data, so the newest values survive. For some data it is the other way round; decide on purpose.
• How does the receiver know what it got? A sequence number per message lets the receiver see gaps and discard duplicates.

from collections import deque

BUFFER_SIZE = 4                     # tiny on purpose; real gateways keep hours
buffer = deque(maxlen=BUFFER_SIZE)  # a full deque drops its OLDEST item
broker_up = {0: True, 1: False, 2: False, 3: False, 4: False, 5: False,
             6: False, 7: True, 8: True}
received = {}                       # what the platform has stored, by sequence

def deliver(message):
    seq, watts = message
    if seq in received:             # duplicate after a retry: ignore it
        return
    received[seq] = watts

for seq in range(9):                # one 10-second average per step
    message = (seq, 600 + 10 * seq)
    if broker_up[seq]:
        replayed = []
        while buffer:               # replay the backlog first, oldest first
            old = buffer.popleft()
            deliver(old)
            replayed.append(old[0])
        deliver(message)
        note = f" after replaying {replayed}" if replayed else ""
        print(f"seq {seq}: sent{note}")
    else:
        if len(buffer) == BUFFER_SIZE:
            print(f"seq {seq}: buffer full, dropping seq {buffer[0][0]}")
        else:
            print(f"seq {seq}: broker down, buffered")
        buffer.append(message)

deliver((7, 670))                   # a retry delivers seq 7 a second time
missing = sorted(set(range(9)) - set(received))
print("stored:", sorted(received))
print("missing:", missing)

seq 0: sent
seq 1: broker down, buffered
seq 2: broker down, buffered
seq 3: broker down, buffered
seq 4: broker down, buffered
seq 5: buffer full, dropping seq 1
seq 6: buffer full, dropping seq 2
seq 7: sent after replaying [3, 4, 5, 6]
seq 8: sent
stored: [0, 3, 4, 5, 6, 7, 8]
missing: [1, 2]

The buffer of four could not cover an outage of six steps, so the two oldest messages were dropped, and the sequence numbers make that visible instead of silent. The duplicate of seq 7 is ignored because the receiver remembers what it has stored: an idempotent consumer. Duplicates are normal with store and forward: if the connection breaks after a message arrived but before the gateway learned that, the gateway sends it again. MQTT's quality-of-service levels make the same trade-off between losing and duplicating messages.

Local-first control

Local-first is a design principle in which a system does its core job with local devices and local data, and treats the cloud as optional: useful for remote access, backups and extra features, but never required for the basic function. It is the edge-computing answer to the autonomy question: what still works when the internet is down?

Lotte's battery automation is local-first. The prices for the next day are known in advance, so Home Assistant fetches them once and keeps them; the decision to charge or discharge runs on the Pi, using the live P1 power and the battery's own interface on the home network. If her internet connection fails at 18:00, the battery still discharges at the evening peak, because nothing in that loop depends on a server outside the house. Only the remote app view stops working. The design choices that make this true:

• Devices talk to a local broker or hub, not directly to a vendor's cloud.
• Data the decision needs, such as prices and schedules, is cached locally with a sensible fallback when it is stale.
• Every automated device has a safe default and a manual override: a battery that has lost its controller should stop, not keep charging.

The Kestrel Home pilot is deliberately not local-first: its job is to show live power in the Kestrel app, which needs Kestrel's platform. That is fine, because nothing in the home depends on it. If Kestrel's broker is down, the app shows no live value and the house carries on.

Edge or cloud?

Most systems split the work. A rough guide for each function:

A common split: the edge filters, aggregates, buffers and runs local control; the cloud stores long histories, compares sites, trains models and manages the fleet (see IoT Platforms and the Cloud). A model trained in the cloud can then run at the edge.

Fog computing

Fog computing is a name for a layer of computing between the devices and the cloud, such as local servers or network equipment that serve many devices in an area. The term is usually credited to Cisco, around 2012. In practice the boundary with edge computing is blurry and most people today say "edge" for both; you will meet "fog" mainly in older material and in research papers.

Running gateways in practice

An edge gateway is a server in an awkward place: in a meter cupboard, on a factory wall, on a roof. It needs everything a server in a data center needs, with nobody nearby to press a button. The points that cause most trouble:

• Updates. A gateway runs an operating system, a broker and several services, each with its own security updates. Automate them, and keep the ability to roll back (see Provisioning and Over-the-Air Updates).
• The clock. Timestamps set at the edge are only as good as the edge clock. Use NTP, check it after power cuts, and keep the measured time when you forward data, so buffered values keep their real time.
• Storage wear. Writing every second to an SD card wears it out. Write aggregates, batch writes, or use an SSD.
• Monitoring. Let the gateway report its own health (uptime, buffer fill level, CRC errors, last message per device), and alert on silence, not only on errors.
• Backups. Configuration, automations and credentials take hours to rebuild by hand. Back them up off the device.
• Security. The gateway holds the credentials for the platform and can reach every local device. Change default passwords, expose no management interface to the internet, and keep it on a separate network segment where possible (see IoT Security).