Firmware and Embedded Programming

What makes embedded programming different

Firmware is the software stored in a device's own flash memory. On a PC, firmware only starts the operating system; on a microcontroller, the firmware is the whole program. Embedded programming is writing that software: programs that run directly on a microcontroller in an embedded system, close to the hardware, with tight limits on memory, energy and timing, and without a person at a keyboard to restart them.

Compared with writing a script or a web service, five things change:

• Little memory. An ESP32 has 520 KB of RAM, a Raspberry Pi Pico 264 kB, a classic Arduino Uno 2 kB. There is no swap and usually no memory protection: running out means a crash.
• Direct hardware access. The program sets bits in the chip's registers to switch pins, start a UART or read the ADC, often through a framework but sometimes directly.
• No operating system, or a very small one. There is no shell, no package manager and no process isolation; at most a real-time operating system that schedules a few tasks.
• Cross-compiling. You build on a laptop for a different processor (Xtensa, RISC-V or ARM) and copy the result to the device.
• It must run forever on its own. A device in a meter cupboard has nobody to press Ctrl+C. It must survive bad input, lost Wi-Fi and its own bugs, and recover by itself.

Languages and frameworks

C is the main language of firmware: compilers exist for every microcontroller, it gives exact control over memory and registers, and every vendor's examples use it. C++ is common too, often as "C with classes". Around the language, a framework supplies drivers, a build system and libraries.

ESP-IDF

ESP-IDF (Espressif IoT Development Framework) is Espressif's official framework for its ESP32 chips: C libraries for Wi-Fi, Bluetooth, the peripherals, secure boot, flash encryption and over-the-air updates, a build tool, and a version of FreeRTOS extended for the ESP32's two cores. As of October 2026 the latest release is v6.1 (27 August 2026); each release is supported for 30 months, so a product team picks a release and plans its upgrade before support ends. ESP-IDF is what Espressif recommends for products; the other frameworks below often build on it.

The Arduino framework

The Arduino framework hides the hardware behind simple functions such as pinMode(), digitalWrite() and Serial.print(), and runs on many boards, including the ESP32. A program, called a sketch, has two functions: setup() runs once, loop() runs over and over. Thousands of libraries for sensors and displays make it the fastest way from idea to working prototype. Arduino IDE 2.3.10 (June 2026) is the current editor as of October 2026; many developers use the framework from other editors too.

MicroPython

MicroPython is a compact implementation of Python 3 that runs directly on microcontrollers such as the ESP32 family and the Raspberry Pi Pico. It includes a subset of the standard library and a machine module for pins, buses, timers and sleep. You type commands at an interactive prompt (the REPL) over USB and see the result at once, and you copy .py files to the board without compiling. As of October 2026 the current version is v1.29.0 (24 August 2026).

The price is speed and memory: interpreted Python is many times slower than C and needs more RAM, and time-critical work (fast bit-banging, tight timing) belongs in C. For a reader that handles one telegram per second, MicroPython is more than fast enough. Modules such as machine, network and umqtt.simple exist only on MicroPython, not in the Python on your laptop.

Other options exist, such as the Zephyr real-time operating system (the Arduino UNO Q runs its sketches on Zephyr) and configuration-based firmware such as ESPHome, where you describe the device in YAML instead of writing code (see Smart Home and Building Automation).

Building and flashing

A compiler for the target processor (a cross-compiler) turns the source into a binary image on your laptop. Flashing is writing that image into the microcontroller's flash memory, usually over USB or a serial connection, sometimes through a dedicated debug adapter. On an ESP32 development board the USB port carries the serial connection; the flashing tool puts the chip into download mode, writes the image and restarts it.

Typical commands (not run here; the serial port name depends on the board and the computer):

# ESP-IDF: build, flash and open the serial monitor
idf.py -p /dev/ttyUSB0 flash monitor

# MicroPython: copy a program to the board and open the REPL
mpremote connect /dev/ttyUSB0 cp main.py :
mpremote connect /dev/ttyUSB0 repl

# MicroPython: install a library from the micropython-lib index
mpremote mip install umqtt.simple

On Linux the board appears as /dev/ttyUSB0 or /dev/ttyACM0. "Permission denied" on that device usually means your user is not in the group that owns serial ports (dialout on Debian and Raspberry Pi OS). If the port does not appear at all, try another USB cable: many cheap cables carry only power, no data.

The bootloader

A bootloader is the small program that runs first when a device starts and decides which firmware to load and start; it can often also receive new firmware. On an ESP32 the chain has two steps: a first-stage bootloader in the chip's read-only memory, which nobody can change, loads a second-stage bootloader from flash, which reads the partition table and starts the application. With over-the-air updates there are two application slots, and the bootloader picks the active one; with secure boot it first checks the application's signature.

How the two slots make updates safe, including rollback when a new version fails, is explained on Provisioning and Over-the-Air Updates. Secure boot on PCs is on BIOS, UEFI and Boot; the idea is the same.

The super loop

The simplest firmware structure is the super loop: after a one-time setup, the program runs one endless loop that checks every input and handles every job in turn, forever. Arduino's setup() and loop() are exactly this. Hardware events that cannot wait, such as a pin change, are handled by interrupt service routines that set a flag for the loop.

The rule that keeps a super loop working: no step may block for long. A delay(500) to blink an LED stops everything else for half a second, including reading the UART, whose bytes then overflow. Instead, check the time and act when it is due:

const int LED_PIN = 2;             // pin number depends on the board
unsigned long lastBlink = 0;
bool ledOn = false;

void setup() {
  pinMode(LED_PIN, OUTPUT);
  Serial.begin(115200);
}

void loop() {
  readP1Bytes();                         // your function: take what has arrived, never wait

  if (millis() - lastBlink >= 500) {     // instead of delay(500)
    lastBlink = millis();
    ledOn = !ledOn;
    digitalWrite(LED_PIN, ledOn ? HIGH : LOW);
  }
}

A super loop is easy to understand and debug, and enough for many devices. It becomes awkward when jobs have very different timing, or when a library (such as a Wi-Fi stack) needs to run in the background. That is where a real-time operating system comes in.

Real-time operating systems and FreeRTOS

A real-time operating system (RTOS) lets firmware run several tasks that each look like their own endless loop. A scheduler, driven by a regular tick interrupt, switches between them by priority, and the tasks exchange data through queues and protect shared data with mutexes and semaphores. The usual problems of concurrency apply: two tasks writing the same variable without a lock is a race condition, on a microcontroller as on a server.

FreeRTOS is the most widely used open-source real-time operating system kernel for microcontrollers: a small library that provides tasks, queues, semaphores and mutexes, software timers and a tick-based scheduler. It is MIT-licensed (since version 10.0.0) and maintained under Amazon Web Services' stewardship; as of October 2026 the current kernel is V11.3.1 (August 2026). ESP-IDF includes its own variant, based on FreeRTOS 10.5.1 with significant changes to schedule tasks on the ESP32's two cores. On an ESP32, Wi-Fi and Bluetooth already run as FreeRTOS tasks, even when your own code is a single loop.

A P1 reader split into two tasks: one reads and checks telegrams, the other sends them, and a queue sits between them so that a slow network does not hold up the UART (a sketch, not run here):

#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/queue.h"

struct reading { int32_t watts; };
static QueueHandle_t readings;

static void p1_task(void *arg)
{
    struct reading r;
    for (;;) {
        /* wait for a telegram on the UART, check its CRC, fill r */
        xQueueSend(readings, &r, portMAX_DELAY);
    }
}

static void publish_task(void *arg)
{
    struct reading r;
    for (;;) {
        if (xQueueReceive(readings, &r, portMAX_DELAY)) {
            /* publish r.watts over MQTT */
        }
    }
}

void app_main(void)
{
    readings = xQueueCreate(4, sizeof(struct reading));
    xTaskCreate(p1_task, "p1", 4096, NULL, 5, NULL);
    xTaskCreate(publish_task, "publish", 4096, NULL, 4, NULL);
}

Each task gets its own fixed stack (here 4096). Too small and the task overwrites memory or crashes; too large and RAM runs out. Tuning stack sizes is a normal part of RTOS work.

Working with very little memory

On a server, memory is allocated and freed all the time and the operating system cleans up after a process ends. Firmware runs for years in a few hundred kilobytes, so it follows stricter habits:

• Fixed-size buffers, sized at compile time. You know the maximum telegram length, so reserve that much once.
• No allocation in the main loop. Repeatedly allocating and freeing memory on the heap slowly fragments it, until a request fails after weeks of running. A memory leak that costs 100 bytes per message ends the device within a day.
• Integers instead of floating point where the chip has no floating-point unit or where exact values matter: store 1.234 kW as 1234 W.
• Decide what happens when a buffer is full. Dropping data and counting it is better than crashing; the counter tells you the buffer is too small.

Two building blocks appear in almost every firmware. The first is setting and clearing single bits in a hardware register with bit masks. In real firmware the register is a fixed memory address accessed through a volatile pointer (or a framework function); here an ordinary variable stands in for it:

#include <stdint.h>
#include <stdio.h>

#define LED_PIN   2u   /* bit numbers in the output register (illustrative) */
#define RELAY_PIN 5u

static void show(const char *step, uint32_t reg)
{
    printf("%-22s", step);
    for (int bit = 7; bit >= 0; bit--) {        /* print the low 8 bits */
        putchar((reg >> bit) & 1u ? '1' : '0');
    }
    putchar('\n');
}

int main(void)
{
    uint32_t out = 0;                  /* stands in for a GPIO output register */

    show("start", out);
    out |= (1u << LED_PIN);            /* set one bit, leave the rest alone */
    show("LED on", out);
    out |= (1u << RELAY_PIN);
    show("relay on", out);
    out &= ~(1u << LED_PIN);           /* clear one bit */
    show("LED off", out);
    out ^= (1u << RELAY_PIN);          /* toggle one bit */
    show("relay toggled", out);
    printf("relay is %s\n", (out & (1u << RELAY_PIN)) ? "on" : "off");
    return 0;
}

start                 00000000
LED on                00000100
relay on              00100100
LED off               00100000
relay toggled         00000000
relay is off

The second is a ring buffer: a fixed array used as a queue whose read and write positions wrap around. The UART interrupt puts received bytes in; the main loop takes them out when it has time. The buffer here has room for 8 bytes and 11 arrive before the loop reads:

#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>

#define SIZE 8u                        /* fixed at compile time: no malloc */

static uint8_t buffer[SIZE];
static unsigned head, tail, count, dropped;

static bool push(uint8_t byte)         /* called for every byte the UART receives */
{
    if (count == SIZE) {
        dropped++;                     /* full: count the loss instead of crashing */
        return false;
    }
    buffer[head] = byte;
    head = (head + 1) % SIZE;
    count++;
    return true;
}

static bool pop(uint8_t *byte)         /* called by the main loop when it has time */
{
    if (count == 0) {
        return false;
    }
    *byte = buffer[tail];
    tail = (tail + 1) % SIZE;
    count--;
    return true;
}

int main(void)
{
    const char *incoming = "/XYZ5\\2DEMO";   /* 11 bytes arrive before the loop reads */
    uint8_t byte;

    for (const char *p = incoming; *p != '\0'; p++) {
        push((uint8_t)*p);
    }
    printf("stored %u, dropped %u\n", count, dropped);

    printf("read: ");
    while (pop(&byte)) {
        putchar(byte);
    }
    putchar('\n');
    return 0;
}

stored 8, dropped 3
read: /XYZ5\2D

Three bytes are lost, and the telegram they belonged to will fail its CRC check. At 115200 baud a 631-byte telegram arrives in about 55 ms, so the receive buffer must hold a whole telegram or the program must read it out faster than it arrives. That is why P1 reader software typically configures a large receive buffer.

Watchdog timers

A watchdog timer is a hardware timer that resets the microcontroller unless the firmware restarts ("feeds" or "kicks") it before it runs out. If the program hangs in an endless loop, waits forever for a network that never answers or crashes in a way that leaves the chip running, the watchdog restarts it. For a device nobody will visit, it is the last line of defense.

The trick is where you feed it. Feeding it at the top of the main loop only proves that the loop runs. Feed it only after the device has done useful work, such as reading a valid telegram, and the watchdog also catches the case where the loop runs but nothing works. The simulation below follows that rule with a 30-second timeout; the P1 cable is pulled out after 10 seconds:

WATCHDOG_TIMEOUT_S = 30


def telegram_ok(second):
    """Simulated P1 input: valid telegrams until the cable is pulled at 10 s."""
    return second < 10


last_feed = 0
for second in range(60):                 # one pass of the super loop per second
    if telegram_ok(second):
        last_feed = second               # feed only after real progress
        if second in (0, 9):
            print(f"{second:>2} s  telegram OK, watchdog fed")
    if second - last_feed >= WATCHDOG_TIMEOUT_S:
        print(f"{second:>2} s  no valid telegram for {WATCHDOG_TIMEOUT_S} s: "
              "watchdog resets the board")
        break
print("after the reset: boot, reconnect Wi-Fi, try the P1 port again")

 0 s  telegram OK, watchdog fed
 9 s  telegram OK, watchdog fed
39 s  no valid telegram for 30 s: watchdog resets the board
after the reset: boot, reconnect Wi-Fi, try the P1 port again

ESP-IDF has two built-in watchdogs: the interrupt watchdog checks that interrupt handlers do not block the processor too long, and the task watchdog checks that tasks keep giving the processor back to the scheduler. MicroPython offers machine.WDT, with a minimum timeout of one second on the ESP32. In MicroPython a watchdog cannot be stopped once it runs, so it keeps protecting the device even when the code that should feed it is stuck.

Putting it together: Lotte's P1 reader

Lotte Kortenhoeve, a Kestrel Energy customer in Utrecht, wrote the first version of her smart-meter reader in MicroPython on an ESP32 development board. It reads the inverted UART of the P1 port (see Bus Protocols), checks each telegram's CRC (the function and the format are explained on Smart Meters and the P1 Port), and publishes the current power to the MQTT broker on her Raspberry Pi (see MQTT and IoT Messaging). The watchdog resets the board when no valid telegram has arrived for 30 seconds. Wi-Fi setup is left out; pin numbers and the broker address are examples, and keyword arguments can differ between ports, so check the MicroPython documentation for your board.

from machine import UART, WDT
from umqtt.simple import MQTTClient

uart = UART(1, baudrate=115200, bits=8, parity=None, stop=1,
            rx=16, timeout=2000, invert=UART.INV_RX)
wdt = WDT(timeout=30000)                  # reset after 30 s without feed()
mqtt = MQTTClient("p1-reader", "192.168.1.20")
mqtt.connect()


def crc16_arc(data):
    crc = 0
    for byte in data:
        crc ^= byte
        for _ in range(8):
            crc = (crc >> 1) ^ 0xA001 if crc & 1 else crc >> 1
    return crc


def read_telegram():
    """Collect lines from "/" up to and including the "!CRC" line."""
    lines = []
    while True:
        line = uart.readline()            # None after the timeout
        if line is None:
            return None
        if line.startswith(b"/"):
            lines = []
        lines.append(line)
        if line.startswith(b"!"):
            return b"".join(lines)


def is_valid(telegram):
    end = telegram.index(b"!") + 1
    try:
        sent = int(telegram[end:].strip(), 16)    # also accepts an unpadded CRC
    except ValueError:                            # garbled CRC field: drop the telegram
        return False
    return crc16_arc(telegram[:end]) == sent


def import_watts(telegram):
    for line in telegram.split(b"\r\n"):
        if line.startswith(b"1-0:1.7.0("):
            return round(float(line[10:line.index(b"*")]) * 1000)
    return None


while True:
    telegram = read_telegram()
    if telegram and is_valid(telegram):
        wdt.feed()                        # fed only after a valid telegram
        watts = import_watts(telegram)
        if watts is not None:
            mqtt.publish(b"home/p1/power", str(watts).encode())

The program has the shape of every small device: read, validate, act, and let the watchdog catch whatever goes wrong. It also publishes a bare number of watts; a later version adds the unit and the measurement time to every message (see MQTT and IoT Messaging). What it does not yet handle well is a lost connection to the broker: publish() then raises an exception that stops the program, and the watchdog resets the board 30 seconds later. That works, but reconnecting with a delay would be kinder to the broker and the network; see Edge Computing and Gateways for buffering readings while the broker is away.

Debugging firmware

The serial console is the first tool: everything the firmware prints arrives on your laptop over the USB cable, together with the messages of the framework and the bootloader. Connecting to a MicroPython board shows its prompt:

MicroPython v1.29.0 on 2026-08-24; Generic ESP32 module with ESP32
Type "help()" for more information.
>>>

Things to look for when a device misbehaves:

• The reset reason printed at start-up, or read in code (machine.reset_cause() in MicroPython): power-on, watchdog, brownout (supply voltage too low; see Power and Batteries) or a crash.
• A reset loop: the device restarts every few seconds. Often a crash early in start-up, a watchdog that fires during a slow step, or a power supply that collapses when Wi-Fi starts.
• Free memory over time: print it every hour. A number that only goes down is a leak.
• Counters: telegrams read, CRC errors, bytes dropped, reconnects. They turn "it sometimes fails" into a number you can follow.