Provisioning and Over-the-Air Updates

The device lifecycle

A server lives in a data center that a team can walk into. An IoT device lives in someone's meter cupboard for five or ten years, and after it leaves the factory nobody from the company may ever touch it again. Everything it will need to be trusted, configured and repaired has to be possible remotely. That is why the lifecycle starts in the factory and ends with a deliberate step, not with a device that is simply forgotten.

Device provisioning is the process that turns a blank device into a known member of a fleet: it receives its identity (keys and a certificate), it is registered in the platform's device registry, it gets the settings it needs to connect (a Wi-Fi network, the broker's address) and it is linked to an owner or location. The term "provisioning" is also used for creating user accounts (see Identity Lifecycle) and for creating infrastructure; the device sense is closest to the first, but for things instead of people.

This page follows the Kestrel Home pilot, which started in September 2026: about 250 households with a dynamic contract received a P1 dongle, built around an ESP32-C3 by an outside manufacturer and sold under Kestrel's name. With the customer's explicit consent, the dongle reads the P1 port of the smart meter and sends live power to Kestrel's MQTT broker over TLS, so the Kestrel app can show live use and battery advice. One of the pilot homes is Lotte Kortenhoeve's, home h0017. The pilot's design, MQTT over TLS with a certificate per dongle, is written down in ADR-0011 (proposed).

Device identity

A device identity is what a device uses to prove who it is to the platform, and what the platform uses to decide what that device may do. In practice it is a unique key pair per device plus an X.509 certificate that binds the public key to a device name, signed by a certificate authority the platform trusts. The device then connects with mutual TLS: the broker shows its certificate to the device, and the device shows its own to the broker.

Why a certificate per device instead of one shared secret:

• A stolen device is one device. Anyone can open a dongle and read its flash. If all 250 dongles shared one password, reading one would let an attacker pretend to be all of them. With a key per device, Kestrel revokes one certificate and the rest of the fleet is untouched.
• Authorization can follow identity. The broker lets the dongle with certificate KHP1-000231 publish only to the topics of the home it is paired with.
• Logs mean something. Every connection, every update and every failure can be traced to one physical device.

The private key should never leave the device. The best place for it is a secure element or a chip feature that keeps keys out of reach of the firmware; the minimum is encrypted flash. The key should be generated on the device itself, during production, so that it never exists anywhere else.

A pilot dongle has two identifiers that are easy to mix up: its serial (KHP1-000231, fixed in the factory and in its certificate) and the home it is paired with (h0017, assigned when the customer pairs it). If Lotte's dongle breaks and is replaced, the home stays and the serial changes. The meter has its own identifiers again (see Smart Meters and the P1 Port). A certificate check from the service desk could look like this (illustrative):

openssl x509 -in KHP1-000231.pem -noout -subject -issuer -dates

subject=CN = KHP1-000231, O = Kestrel Energy
issuer=CN = Kestrel Home Device CA 1, O = Kestrel Energy
notBefore=Aug 18 09:12:44 2026 GMT
notAfter=Aug 15 09:12:44 2036 GMT

Ten years is longer than a web server certificate would ever get (see certificate lifetimes), because a device that is offline or forgotten cannot renew on time. Long validity is a trade-off, not a free choice: it only works if revocation works, and the fleet still needs a way to rotate certificates if the CA itself must be replaced.

Provisioning at scale

Setting up one device by hand takes a few minutes. Setting up 250, or 250,000, by hand is not an option. Fleet provisioning is provisioning designed for many devices at once: identities are created and registered automatically, from the production line or on the device's first connection, with no per-device manual steps. There are three common patterns.

Provisioning in the factory

Each device generates its key pair on the production line and sends a certificate signing request to the customer's device CA; the signed certificate is written back into the device, and the serial and certificate fingerprint go into a file that is imported into the device registry. This is how the Kestrel Home dongles were made. It needs trust in the factory, but the device is ready the moment it is unpacked.

Provisioning by claim

All devices of a batch ship with the same, limited claim certificate, which may do only one thing: request a unique certificate on first connection. The platform checks the request (for example against a list of expected serials), issues the device its own certificate and registers it. AWS IoT Core calls this "fleet provisioning by claim" (see IoT Platforms and the Cloud). The claim key is shared, so it must be useless for anything else and easy to retire.

Onboarding in the home

The last step happens at the customer: the device must learn the home's Wi-Fi network and be linked to an account. Common methods are a temporary Wi-Fi access point with a small setup page, or Bluetooth Low Energy from a phone app. For the pilot, Lotte plugged the dongle into the P1 port and the Kestrel app connected to the dongle's temporary setup hotspot, using the password printed on its label; she entered her Wi-Fi password and confirmed her consent, the app linked serial KHP1-000231 to home h0017 in the registry, and the hotspot switched off. The service desk's article KB-0610, "Connecting the Kestrel Home dongle to the P1 port", walks through the same steps. Smart home devices that follow Matter use a standardized version of this onboarding (see Smart Home and Building Automation).

The resulting registry entry might look like this (illustrative):

{
  "serial": "KHP1-000231",
  "model": "Kestrel Home P1 dongle, ESP32-C3",
  "certificate_sha256": "3F:9A:1C:...:E7",
  "home": "h0017",
  "firmware": "1.4.0",
  "consent": "live-power, given 2026-09-14",
  "status": "active"
}

Over-the-air updates

An over-the-air update (OTA update) is a firmware update that a device downloads and installs over its network connection, without anyone connecting a cable. For a fleet in other people's homes it is the only realistic way to fix bugs and security holes. It is also what security standards and law now expect: ETSI EN 303 645 has a provision "Keep software updated", and the EU Cyber Resilience Act requires security updates for products with digital elements; UK law requires a published minimum update period for consumer connectable products (see IoT Security).

The general idea of updating firmware and its risks are on BIOS, UEFI and Boot. What changes for IoT is that nobody stands next to the device. A PC whose BIOS update fails can be repaired by a technician; a dongle in a meter cupboard that no longer boots is a lost device and an unhappy customer. An OTA update therefore follows a fixed sequence:

1. The device learns that a new version exists (a message from the platform, or a periodic check).
2. It downloads the image over TLS and checks its signature, so only firmware signed by the manufacturer is accepted (firmware signing and secure boot are on IoT Security).
3. It writes the image to flash that is not in use, while the old firmware keeps running.
4. Only after the complete image is written and verified does it mark the new image as the one to boot, and restart.
5. The new firmware tests itself and confirms, or the device goes back to the old version.

A/B partitions

Steps 3 to 5 need room for two complete firmware images. A/B partitioning divides the flash into two application slots, A and B, of which one runs while the other receives the update; a small record tells the bootloader which slot to start. If the power fails halfway through a download, the running slot is untouched and the device simply tries again later. On the ESP32 family the slots are called ota_0 and ota_1 and the record is the otadata partition.

The layout is defined in a partition table. For a dongle with 4 MB of flash it could look like this (illustrative; offsets and sizes in hexadecimal, app slots aligned to 64 KB):

# Name,   Type, SubType, Offset,   Size
nvs,      data, nvs,     0x9000,   0x4000
otadata,  data, ota,     0xd000,   0x2000
phy_init, data, phy,     0xf000,   0x1000
ota_0,    app,  ota_0,   0x10000,  0x1E0000
ota_1,    app,  ota_1,   0x1F0000, 0x1E0000

Each slot holds 0x1E0000 bytes (1,966,080 bytes, about 1.9 MiB), so a firmware image must stay below that, with room to grow over the years. Doubling the flash for a second slot costs money on every device; it is still far cheaper than one truck roll.

The switch itself must also survive a power cut. ESP-IDF keeps two copies of otadata in two flash sectors, each with a sequence number and a check value, and an update rewrites only the older copy. The bootloader trusts the valid copy with the highest sequence number. Simplified, in C, the language bootloaders are written in:

#include <stdint.h>
#include <stdio.h>

#define APP_SLOTS 2

/* One copy of the OTA data. The flash keeps two copies in two sectors and
   an update rewrites only the older one. */
struct otadata {
    uint32_t seq;   /* sequence number, raised by every switch */
    uint32_t check; /* check value over seq (a CRC-32 in ESP-IDF) */
};

static uint32_t check_value(uint32_t seq) /* stand-in for the CRC */
{
    return seq * 2654435761u;
}

static int is_valid(const struct otadata *copy)
{
    return copy->seq != 0xFFFFFFFFu && copy->check == check_value(copy->seq);
}

/* What the bootloader does: trust the valid copy with the highest
   sequence number, then boot slot (seq - 1) % APP_SLOTS. */
static void choose(const char *situation, const struct otadata *a,
                   const struct otadata *b)
{
    uint32_t seq = 0;
    if (is_valid(a) && a->seq > seq) seq = a->seq;
    if (is_valid(b) && b->seq > seq) seq = b->seq;
    if (seq == 0) {
        printf("%-22s -> no valid OTA data\n", situation);
        return;
    }
    printf("%-22s -> boot ota_%u (seq %u)\n", situation,
           (unsigned)((seq - 1) % APP_SLOTS), (unsigned)seq);
}

int main(void)
{
    struct otadata a = {5, check_value(5)};
    struct otadata b = {4, check_value(4)};
    choose("running normally", &a, &b);

    b.seq = 6;              /* power fails halfway through the write: */
    choose("copy B half-written", &a, &b); /* new seq, old check value */

    b.check = check_value(6);
    choose("copy B written", &a, &b);
    return 0;
}

running normally       -> boot ota_0 (seq 5)
copy B half-written    -> boot ota_0 (seq 5)
copy B written         -> boot ota_1 (seq 6)

The half-written copy fails its check, so the bootloader ignores it and starts the old slot, exactly as if the update had never happened. The real bootloader uses a CRC-32 instead of the stand-in function (see cyclic redundancy checks), but the decision is the same. Erased flash reads as all ones, which is why 0xFFFFFFFF counts as "empty".

Self-tests and rollback

A/B partitions protect against an interrupted download. They do not protect against firmware that installs perfectly and then does not work: it crashes at startup, cannot read the meter, or, worst of all, can no longer connect to the update server, so it can never be fixed remotely. Firmware rollback is returning a device to the previous firmware version when a new one fails. Done well, it happens automatically on the device, without any message from the platform: the new image starts on probation, and if it does not confirm that it works, the bootloader goes back.

ESP-IDF supports this with an option for app rollback (CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE). A newly installed image starts in the state NEW, becomes PENDING_VERIFY on its first boot, and must then call a function to become VALID. If the device resets while the image is still pending, for example because a watchdog timer fired, the bootloader marks it ABORTED and starts the previous slot. A simplified model of those states:

# A simplified A/B update with automatic rollback, using the image
# states of ESP-IDF's app rollback feature.
OTHER = {"ota_0": "ota_1", "ota_1": "ota_0"}


class Dongle:
    def __init__(self, version):
        self.image = {"ota_0": version, "ota_1": "-"}   # firmware in each slot
        self.state = {"ota_0": "VALID", "ota_1": "-"}   # image states (otadata)
        self.boot = "ota_0"                             # slot that boots next

    def show(self, event):
        print(f"{event:<20} boot={self.boot}  "
              f"0: {self.image['ota_0']} {self.state['ota_0']:<14} "
              f"1: {self.image['ota_1']} {self.state['ota_1']}")

    def install(self, version):
        target = OTHER[self.boot]                       # never the running slot
        self.image[target], self.state[target] = version, "NEW"
        self.boot = target                              # only after a verified write
        self.show(f"install {version}")

    def reboot(self, why="reboot"):
        if self.state[self.boot] == "NEW":
            self.state[self.boot] = "PENDING_VERIFY"    # first boot of the image
        elif self.state[self.boot] == "PENDING_VERIFY":
            self.state[self.boot] = "ABORTED"           # it never confirmed
            self.boot = OTHER[self.boot]                # bootloader rolls back
        self.show(why)

    def self_test(self, passed):
        if passed:
            self.state[self.boot] = "VALID"             # cancel the rollback
            self.show("self-test passed")
        else:
            self.state[self.boot] = "INVALID"
            self.boot = OTHER[self.boot]                # mark invalid, reboot
            self.show("self-test failed")


dongle = Dongle("1.3.2")
dongle.show("start")
dongle.install("1.4.0")
dongle.reboot()
dongle.self_test(passed=True)
print("-- an update that hangs before its self-test")
dongle.install("1.4.1")
dongle.reboot()
dongle.reboot("watchdog reset")
print("-- an update whose self-test fails")
dongle.install("1.4.2")
dongle.reboot()
dongle.self_test(passed=False)

start                boot=ota_0  0: 1.3.2 VALID          1: - -
install 1.4.0        boot=ota_1  0: 1.3.2 VALID          1: 1.4.0 NEW
reboot               boot=ota_1  0: 1.3.2 VALID          1: 1.4.0 PENDING_VERIFY
self-test passed     boot=ota_1  0: 1.3.2 VALID          1: 1.4.0 VALID
-- an update that hangs before its self-test
install 1.4.1        boot=ota_0  0: 1.4.1 NEW            1: 1.4.0 VALID
reboot               boot=ota_0  0: 1.4.1 PENDING_VERIFY 1: 1.4.0 VALID
watchdog reset       boot=ota_1  0: 1.4.1 ABORTED        1: 1.4.0 VALID
-- an update whose self-test fails
install 1.4.2        boot=ota_0  0: 1.4.2 NEW            1: 1.4.0 VALID
reboot               boot=ota_0  0: 1.4.2 PENDING_VERIFY 1: 1.4.0 VALID
self-test failed     boot=ota_1  0: 1.4.2 INVALID        1: 1.4.0 VALID

Versions 1.4.1 and 1.4.2 are invented faulty builds; in both cases the device ends up on 1.4.0 again without anyone touching it. In real ESP-IDF firmware the check sits at the start of the application (not run here; the self-test function is the firmware's own):

const esp_partition_t *running = esp_ota_get_running_partition();
esp_ota_img_states_t state;

if (esp_ota_get_state_partition(running, &state) == ESP_OK &&
    state == ESP_OTA_IMG_PENDING_VERIFY) {
    if (self_test_passed()) {   /* Wi-Fi, broker over TLS, one valid telegram */
        esp_ota_mark_app_valid_cancel_rollback();
    } else {
        esp_ota_mark_app_invalid_rollback_and_reboot();
    }
}

What the self-test checks decides how much the rollback is worth. For the pilot dongle, version 1.4.0 confirms only after it has joined Wi-Fi, connected to the broker over TLS with its own certificate, and read one P1 telegram with a correct CRC. Each of those is something a bad build could break, and the second is the one that matters most: a device that cannot reach its broker can never receive the fix.

Staged rollouts

Rollback protects one device. A staged rollout protects the fleet: a new firmware version goes to a small share of devices first, then to larger shares, with a check between the waves and a rule that stops the rollout when too many devices fail. The first small wave is a canary release; the difference with a web service is that every device updates separately, and some are switched off or offline for days.

Firmware 1.4.0 for the pilot dongles goes out under change CHG-5012, coordinated by release manager Yara Haddad. It fixes how the dongle reconnects after an outage. The plan: 5% of the 250 dongles on Thursday 1 October 2026, 25% on Monday 5 October, and the rest on Thursday 8 October; between waves, the confirm rate is checked, and the rollout stops if more than one device in a wave rolls back.

import math

FLEET = 250                       # pilot dongles
WAVES = [0.05, 0.25, 1.00]        # share of the fleet that has 1.4.0 after each wave
MAX_ROLLBACKS = 1                 # halt rule per wave


def plan(fleet, waves):
    done = 0
    for number, share in enumerate(waves, start=1):
        target = math.ceil(fleet * share)
        yield number, share, target - done, target
        done = target


def run(rollbacks_per_wave):
    for number, share, new, total in plan(FLEET, WAVES):
        failed = rollbacks_per_wave[number - 1]
        verdict = "halt" if failed > MAX_ROLLBACKS else "continue"
        print(f"wave {number} ({share:.0%}): {new:3} devices, total {total:3}, "
              f"{failed} rolled back -> {verdict}")
        if verdict == "halt":
            print(f"stopped: {FLEET - total} devices stay on 1.3.2")
            break


run([0, 1, 0])
print()
run([0, 4, 0])

wave 1 (5%):  13 devices, total  13, 0 rolled back -> continue
wave 2 (25%):  50 devices, total  63, 1 rolled back -> continue
wave 3 (100%): 187 devices, total 250, 0 rolled back -> continue

wave 1 (5%):  13 devices, total  13, 0 rolled back -> continue
wave 2 (25%):  50 devices, total  63, 4 rolled back -> halt
stopped: 187 devices stay on 1.3.2

The waves are cumulative: "25%" means 63 devices in total, so the second wave adds 50. Rounding up with math.ceil makes sure even a small share is at least one device. As of 7 October 2026 the first two waves of CHG-5012 have run with one rollback in wave 2, which the halt rule allows; the rollback was a dongle whose home Wi-Fi was down during the self-test. The firmware itself was fine, but the self-test cannot tell a bad build from a bad network, and a device that cannot prove it reaches the broker must not keep a new image. An occasional false alarm like this is the price of that safety, and the dongle can simply be offered 1.4.0 again.

Choosing the first wave is a decision of its own:

• Mix the hardware. Include each hardware revision and each meter type you know of (DSMR 4 and DSMR 5 meters behave differently on the P1 port).
• Prefer devices you can reach. Staff and friendly testers who will report problems are good canaries.
• Count offline devices separately. A dongle that has not checked in has neither confirmed nor failed. A wave is done when its devices have reported, not when the server has offered the update.

Even within a wave, devices should not all start at the same second. 187 dongles that download a 1.2 MB image at the same moment put a load on the update server and on the broker when they all reboot and reconnect. Adding a random delay, the same idea as jitter in retries, spreads the work:

import random
from collections import Counter

DEVICES = 187          # wave 3 of the rollout
WINDOW_MIN = 60        # spread the downloads over one hour
IMAGE_MB = 1.2         # size of the firmware image

random.seed(5012)      # fixed seed: the same "random" result every run
start_minute = [random.randrange(WINDOW_MIN) for _ in range(DEVICES)]
busiest = max(Counter(start_minute).values())

print(f"no spreading: {DEVICES} downloads in minute 0, "
      f"{DEVICES * IMAGE_MB:.0f} MB at once")
print(f"random delay: at most {busiest} downloads start in the same minute, "
      f"{busiest * IMAGE_MB:.1f} MB")

no spreading: 187 downloads in minute 0, 224 MB at once
random delay: at most 7 downloads start in the same minute, 8.4 MB

Why this matters is shown by Kestrel's own reconnect storm, INC-33004, in which all dongles reconnected at once after a broker restart (see MQTT and IoT Messaging).

Device decommissioning

Devices leave a fleet: they break, customers move, contracts end, pilots stop. Device decommissioning is taking a device out of service on purpose, so that it can no longer connect, holds no secrets or personal data, and no longer appears as active in the registry. It is the device version of decommissioning in a data center, with one difference: the device is usually not in your hands, but in the customer's drawer, a recycling bin or a second-hand webshop.

On 7 October 2026 Lotte Kortenhoeve logged REQ-3120: when her own home battery automation is finished at the end of October, she wants to leave the pilot, and she asks Kestrel to delete her pilot data then. ETSI EN 303 645 asks manufacturers to "make it easy for users to delete user data", and the GDPR gives her the right to erasure. The planned steps:

1. Revoke the certificate of KHP1-000231 (see certificate revocation) and block it at the broker, so the dongle cannot connect again, whoever has it.
2. Unlink it in the registry: home h0017 loses its device and the record gets the status "decommissioned", with the date and the request number. Keep the record itself, without personal data, so the serial is never silently reused.
3. Delete the personal data: the live power stored for h0017 and the consent record's data, according to the retention rules agreed with the data protection officer (see retention).
4. Wipe the device: a factory reset that erases the Wi-Fi password and settings from the nvs partition. Lotte can do this in the app; a returned dongle is wiped again before it is refurbished.
5. Confirm to the customer and close the request.

The whole fleet also has an end. In the EU (under the CRA from December 2027) and in the UK (for consumer connectable products, already now) a manufacturer must say for how long a product gets security updates, and the EU Cyber Resilience Act sets a minimum support period. Plan the end before it arrives: what devices do when the platform switches off, and how customers are told (compare end-of-life systems in OT).

When it goes wrong

Most provisioning and update problems reach the service desk as "my dongle is offline". These are the usual causes: