Webhooks and Real-Time APIs

Getting updates: four approaches

HTTP is a request-response protocol: the client asks, the server answers. That works well for "show me my bill", but not for "tell me when the outage in my area is fixed". The server knows the moment something changes; the client does not. There are four ways to close that gap, and they differ in who starts the conversation and how long a connection stays open.

• Polling: the client repeats a normal request on a timer, for example every 30 seconds. Simple, works through every proxy and firewall, and is easy to cache. The cost: most answers are "nothing new", and a change waits until the next poll, so you trade freshness against load.
• Long polling: the client asks, but the server holds the request open until it has news.
• Push to a URL (webhooks): the server calls your server when something happens. This is for server-to-server traffic.
• A connection that stays open: WebSockets (two-way) and server-sent events (server to client). This is how browsers get live updates.

Long polling

With long polling, the client sends an ordinary HTTP request and the server deliberately does not answer straight away. It waits until there is something to report or until a time limit (often 20–60 seconds) runs out, then responds; the client immediately sends the next request. The result feels almost instant and uses plain HTTP, so it works wherever normal requests work. It is a technique, not a standard; RFC 6202 (an informational RFC from 2011) describes its known problems.

Those problems are practical: every waiting request occupies a connection on the server, an answer that arrives just as the client reconnects can be missed unless the client sends a "since" marker (a cursor or the last event ID), and intermediaries with a shorter idle timeout than the server's wait cut requests off. Long polling is a reasonable fallback; for new browser features, server-sent events or WebSockets are usually cleaner.

Webhooks in depth

A webhook turns the API around: instead of you calling the provider, the provider sends an HTTP POST to a URL you registered, as soon as an event happens (the basic idea is on What Is an API?). GitHub calls your build server when someone pushes (see the triggers on GitHub Actions), a payment provider tells your shop that a payment succeeded, and Kestrel Energy's outage system can notify a partner's dispatch tool when a fault is resolved. There is no IETF standard for webhooks; each provider defines its own payloads and headers, but the patterns below are nearly universal.

A delivery is just an HTTP request with a JSON body:

POST /hooks/outages HTTP/1.1
Host: hooks.example.com
Content-Type: application/json
User-Agent: Kestrel-Webhooks/1.0
Webhook-Event: outage.resolved
Webhook-Id: evt_20260921_0042
Webhook-Signature: t=1790000000,v1=8e458e4f456a00…

{"event":"outage.resolved","area":"Rotterdam-Noord"}

The header names here are made up for Kestrel Energy (the Standard Webhooks initiative defines similarly named headers with a different format); GitHub, for example, uses X-GitHub-Event, X-GitHub-Delivery (a unique ID per delivery) and X-Hub-Signature-256. What matters is that a delivery carries an event type, a unique ID and a signature.

Delivery and retries

The provider considers a delivery successful when your endpoint answers with a 2xx status within a time limit (often around ten seconds). Anything else — a 500, a timeout, a TLS error, a DNS failure — counts as failed. Most providers then retry later, with growing intervals (exponential backoff) over hours or days, and eventually disable an endpoint that keeps failing. Some, GitHub among them, do not retry automatically and instead let you redeliver from a log of recent deliveries. Read your provider's documentation: the retry policy decides how much you can lose.

Two rules follow from the time limit:

• Answer first, work later. Verify the signature, store the event (in a database table or a queue), answer 200 or 202, and let a background worker do the slow part. A handler that sends emails before it responds will time out and be retried while it is still busy.
• Do not rely on order. Retries and parallel deliveries mean outage.resolved can arrive before outage.updated. Use a timestamp or version in the payload, or fetch the current state from the provider's API when an event arrives.

At-least-once delivery and idempotent handlers

Imagine your handler processed an event and answered 200, but the answer was lost on the way back. The provider sees a timeout and sends the same event again. Because a sender can never be sure whether a lost answer means "not processed" or "processed", webhook systems choose at-least-once delivery: every event arrives one or more times, as long as your endpoint works again within the provider's retry window, and duplicates are normal.

The receiver's answer is an idempotent handler: processing the same event twice has the same effect as processing it once (the general idea is idempotence). The usual technique is to record each event ID and skip IDs you have already seen, inside the same database transaction as the work itself. "Set the outage status to resolved" is naturally idempotent; "send an SMS to every affected customer" is not, so it needs the ID check. The same problem between your own services, with business keys, dead-letter queues and exactly-once processing, is on Service Communication.

Verifying webhook signatures

A webhook endpoint is a public URL that accepts POST requests. Anyone who finds it can send a fake payment.succeeded. A webhook signature proves two things: the request came from someone who knows the shared secret, and the body was not changed on the way. It is an HMAC (hash-based message authentication code): when you register the endpoint, the provider gives you a secret; for each delivery it computes HMAC-SHA256 over the body with that secret and sends the result in a header. You compute the same value and compare. The general technique of signing requests is covered on API Authentication.

Two common schemes

• GitHub sends X-Hub-Signature-256: sha256=<hex digest>, an HMAC-SHA256 of the raw body. The older X-Hub-Signature header uses SHA-1 and exists only for compatibility; do not use it.
• Stripe-style schemes sign a timestamp together with the body. Stripe sends Stripe-Signature: t=<Unix time>,v1=<signature> and its secrets start with whsec_. Because the timestamp is inside the signed data, an attacker cannot change it without breaking the signature, so the receiver can refuse deliveries older than a few minutes. That stops a replay: someone who captured a genuine delivery (from a log, say) cannot send it again tomorrow.

A signature check in Python

The sender and the receiver below share one secret. The sender signs timestamp + "." + body; the receiver checks the age of the timestamp, recomputes the signature and compares. A body with one changed field, or a genuine delivery replayed an hour later, is rejected.

import hashlib
import hmac

SECRET = b"demo-secret-shared-with-the-sender"  # in real life: from a secrets manager
TOLERANCE = 300  # seconds a delivery may be old before we refuse it


def sign(body, timestamp):
    """What the sender does: sign the timestamp and the raw body together."""
    signed = f"{timestamp}.".encode("utf-8") + body
    digest = hmac.new(SECRET, signed, hashlib.sha256).hexdigest()
    return f"t={timestamp},v1={digest}"


def verify(header, body, now):
    """What the receiver does, before it trusts anything in the body."""
    parts = dict(item.split("=", 1) for item in header.split(","))
    timestamp = int(parts["t"])
    if abs(now - timestamp) > TOLERANCE:
        return "rejected: timestamp too old (possible replay)"
    expected = sign(body, timestamp).split("v1=")[1]
    if not hmac.compare_digest(expected, parts["v1"]):
        return "rejected: signature does not match"
    return "accepted"


body = b'{"event":"outage.resolved","area":"Rotterdam-Noord"}'
sent_at = 1790000000  # Unix time chosen by the sender
header = sign(body, sent_at)
print("Signature header:", header[:30] + "...")

tampered = body.replace(b"Rotterdam-Noord", b"Utrecht-Oost")

print("Original body:         ", verify(header, body, now=sent_at + 2))
print("Tampered body:         ", verify(header, tampered, now=sent_at + 2))
print("Replayed an hour later:", verify(header, body, now=sent_at + 3600))

Signature header: t=1790000000,v1=8e458e4f456a00...
Original body:          accepted
Tampered body:          rejected: signature does not match
Replayed an hour later: rejected: timestamp too old (possible replay)

Details that matter in real code:

• Constant-time comparison. hmac.compare_digest (Python) or crypto.timingSafeEqual (Node.js) takes the same time whether the first or the last character differs. A plain == stops at the first difference, and that timing difference can in principle leak how much of a guessed signature is right.
• Sign and verify the raw bytes. Verify the body exactly as it arrived, before any framework parses and re-serializes the JSON. {"a": 1} and {"a":1} are the same JSON but different bytes, and therefore different signatures.
• Keep the secret secret. Store it in a secrets manager, not in the repository, use a separate secret per endpoint, and rotate it if it leaks. Many providers allow two active secrets during a rotation.
• Reject quietly. Answer 400 or 401 without explaining which check failed, and log the details on your side.

Signatures are not the only layer. HTTPS protects the delivery in transit, and some providers publish the IP ranges they send from, which you can allow in a firewall. An IP allowlist alone is weak (ranges change and are shared by many customers of a cloud provider), so treat it as an extra, never as a replacement for the signature.

WebSockets

A WebSocket is a persistent, full-duplex connection between a client and a server: once it is open, either side can send a message at any moment, without the request-response pattern of HTTP. The protocol is RFC 6455 (December 2011). It uses the URL schemes ws:// and wss:// (WebSocket over TLS; always use this one outside a lab) and the normal ports 80 and 443, which is why it gets through most networks.

The upgrade handshake

A WebSocket starts life as an HTTP/1.1 request that asks the server to switch protocols on the same TCP connection:

GET /outages/live HTTP/1.1
Host: api.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Origin: https://portal.example.com

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

After the 101 Switching Protocols response, HTTP is over: both sides now exchange WebSocket frames (text, binary, and the control frames ping, pong and close) on the same connection. Sec-WebSocket-Accept is the SHA-1 hash of the client's key plus a fixed string from the RFC, base64 encoded. It only proves that the server really speaks WebSocket (and is not, say, a cache replaying an old answer); it is not authentication. Later RFCs (8441 and 9220) define how to run WebSockets over HTTP/2 and HTTP/3, but the HTTP/1.1 upgrade above is still what you will see most.

In a browser, the WebSocket JavaScript object does all this for you. Troubleshooting is easier once you know the shape: in the browser's developer tools a working connection shows status 101 and a "Messages" view; a 400, 403 or 404 instead of 101 means the upgrade itself failed, usually because a proxy did not pass the Upgrade header or the server refused the origin.

Security and when to use WebSockets

The same-origin policy does not stop a page on another site from opening a WebSocket to your server, and the browser may send your cookies along (SameSite=None cookies always; in Firefox and Safari also cookies without a SameSite attribute). A WebSocket server must therefore check the Origin header against an allowlist and authenticate the connection (a session cookie, or a short-lived token sent in the first message) — the same reasoning as for cross-site request forgery. Validate every incoming message as you would a request body, and limit message size and rate.

Use WebSockets when traffic really flows both ways and often: chat, collaborative editing, multiplayer games, live dashboards where the user also sends commands, or a control room that both watches and acknowledges alarms. For "the server tells the browser about changes", server-sent events are simpler. And WebSockets give you a pipe, not an API: you have to design your own message format, request IDs, error handling and reconnection.

Server-sent events

Server-sent events (SSE) are a one-way stream from server to client over an ordinary HTTP response. The client makes a normal GET; the server answers with Content-Type: text/event-stream and then simply never finishes the response, writing a new event whenever something happens. SSE is defined in the WHATWG HTML Standard, and browsers provide the EventSource API for it.

The format is plain UTF-8 text, one field per line:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

: connected

id: 41
event: outage
data: {"area": "Rotterdam-Noord", "status": "investigating"}

id: 42
event: outage
data: {"area": "Rotterdam-Noord", "status": "crew on site"}

• data: carries the content (several data: lines are joined with newlines); event: gives it a type; id: labels it; retry: tells the client how many milliseconds to wait before reconnecting.
• An empty line ends one event.
• A line starting with a colon is a comment. Servers send one every 15–30 seconds as a heartbeat, so proxies do not close an idle stream.

The best feature is free recovery. When the connection drops, EventSource reconnects by itself and sends a Last-Event-ID header with the last ID it received, so the server can resend what was missed. The limits: SSE is server-to-client only (the client sends data with normal requests), text only, and the browser's EventSource cannot set custom headers such as Authorization, so SSE in a browser usually relies on cookies.

Streaming three events in Python

The script below starts a tiny SSE server on the local machine (port 0 means "any free port"), connects to it with http.client, and parses the stream line by line as it arrives. http.server is fine for learning like this; its documentation says it is not recommended for production.

import http.client
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

UPDATES = [
    ("41", {"area": "Rotterdam-Noord", "status": "investigating"}),
    ("42", {"area": "Rotterdam-Noord", "status": "crew on site"}),
    ("43", {"area": "Rotterdam-Noord", "status": "resolved"}),
]


class EventStream(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.send_header("Cache-Control", "no-cache")
        self.end_headers()
        self.wfile.write(b": connected\n\n")  # a comment line: clients ignore it
        for event_id, update in UPDATES:
            message = f"id: {event_id}\nevent: outage\ndata: {json.dumps(update)}\n\n"
            self.wfile.write(message.encode("utf-8"))
            self.wfile.flush()  # send each event now, do not wait for the end

    def log_message(self, format, *args):
        pass  # keep the example output clean


server = ThreadingHTTPServer(("127.0.0.1", 0), EventStream)  # port 0: any free port
threading.Thread(target=server.serve_forever, daemon=True).start()

conn = http.client.HTTPConnection("127.0.0.1", server.server_address[1])
conn.request("GET", "/outages/stream", headers={"Accept": "text/event-stream"})
response = conn.getresponse()
print(response.status, response.reason, "|", response.getheader("Content-Type"))

# Minimal SSE parser: collect fields until an empty line, then handle the event
event = {}
for raw in response:  # reads the stream line by line as it arrives
    line = raw.decode("utf-8").rstrip("\r\n")
    if line.startswith(":"):
        continue
    if line == "":
        if "data" in event:
            update = json.loads(event["data"])
            print(f"event {event['id']} ({event['event']}): {update['area']} -> {update['status']}")
        event = {}
        continue
    field, _, value = line.partition(":")
    event[field] = value.removeprefix(" ")

conn.close()
server.shutdown()

200 OK | text/event-stream
event 41 (outage): Rotterdam-Noord -> investigating
event 42 (outage): Rotterdam-Noord -> crew on site
event 43 (outage): Rotterdam-Noord -> resolved

This toy server sends three events and closes the response; a real one keeps the stream open for as long as the client listens. In a browser the client side is two lines of JavaScript: new EventSource("/outages/stream") and an event listener for outage.

WebTransport

WebTransport is a newer browser API that runs over HTTP/3 (and so over QUIC), with an HTTP/2 fallback that lacks datagrams. It offers several independent streams on one connection plus unreliable datagrams, which suit games, live media and other traffic where a late packet is worse than a lost one. As of 2026 the W3C specification is a Candidate Recommendation (snapshot of 30 July 2026), and browser support is not yet universal. Treat it as one to watch: for most applications today, WebSockets and SSE are the practical choices.

Choosing between them

A useful rule: if the receiver is a server, use webhooks (with a polling job as a safety net that catches anything a failed delivery missed). If the receiver is a browser or app and only listens, use SSE. If it also talks back often, use WebSockets.

Operational concerns

Long-lived connections behave differently from short requests, and most production problems with real-time features come from the infrastructure in between, not from the code.

• Idle timeouts everywhere. Reverse proxies, load balancers, firewalls and NAT devices close connections that are silent for too long, often after 60 seconds or a few minutes. Send heartbeats (SSE comments, WebSocket ping frames) more often than the shortest timeout on the path.
• Proxies must be told. A reverse proxy passes the Upgrade header only when configured to, and may buffer an SSE stream until it is "complete" — which is never. A typical Nginx setup:

location /outages/live {             # WebSocket
    proxy_pass http://realtime_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 300s;          # longer than the heartbeat interval
}

location /outages/stream {           # server-sent events
    proxy_pass http://realtime_backend;
    proxy_http_version 1.1;
    proxy_buffering off;              # pass each event on at once
    proxy_read_timeout 300s;
}

• Connection limits. Each open connection holds a socket and some memory on every hop. Ten thousand dashboards mean ten thousand open connections; plan file-descriptor limits and capacity for that. Over HTTP/1.1, browsers also allow only a few connections per server (six in common browsers), and each SSE stream uses one; HTTP/2 multiplexes streams over one connection and largely removes that problem.
• Scaling out. With several server instances, an event that happens on instance A must reach clients connected to instance B. The usual answer is a publish-subscribe channel or message broker between the instances. A load balancer in front needs to support WebSockets and long-lived connections (see Web Servers and Reverse Proxies).
• Deployments drop connections. Restarting a server disconnects every client at once. Clients must reconnect with exponential backoff and random jitter, or they all return in the same second and overload the new instance.
• Webhook endpoints need monitoring too. Watch the provider's delivery log, alert on repeated failures before the provider disables your endpoint, and keep received events long enough to investigate.