Service Communication

Once a system is split into separately deployed services (see Monolith, Modular Monolith and Microservices), the services need to talk to each other over the network. This page covers the two basic styles and what goes wrong with each. The details of the protocols are elsewhere: REST, gRPC and webhooks on the Web & APIs pages, and the defensive patterns (timeouts, retries, circuit breakers) on Reliability Patterns. The running example is Kestrel Energy: meter-api receives a reading from each of 400,000 smart meters every 15 minutes, and the billing system KBS needs those readings to bill customers.

Synchronous and asynchronous communication

In synchronous communication a service sends a request to another service and waits for the response before it continues, as in an HTTP call or a gRPC call (also called request-response). It is simple to write and to reason about: the answer, or the error, comes back right away.

In asynchronous messaging a service sends a message to a queue or broker and continues without waiting; the receiving service picks the message up and processes it later, when it is ready. If a reply is needed at all, it comes back as another message.

Do not confuse this with asynchronous programming inside one program (async/await, an event loop). That is about not blocking a thread while waiting; the call can still be a synchronous request-response call to another service. Asynchronous messaging is about not needing the other service to be there at all at the moment you send.

Temporal coupling

Temporal coupling is the dependency between two services that exists when an operation can only succeed if both are available and responsive at the same time. Every synchronous call creates it. Suppose meter-api called KBS for every reading, and KBS is down for three minutes of planned maintenance:

# Readings per minute from meter-api; KBS is down for maintenance in minutes 3-5.
ARRIVALS = 1_000          # readings per minute
KBS_CAPACITY = 2_500      # readings per minute KBS can process when it is up
DOWN = {3, 4, 5}

failed = 0                # synchronous: meter-api calls KBS for every reading
backlog = 0               # asynchronous: meter-api puts readings on a queue
print("min  KBS   sync: failed so far   async: waiting in queue")
for minute in range(10):
    up = minute not in DOWN
    if not up:
        failed += ARRIVALS
    backlog += ARRIVALS
    if up:
        backlog -= min(backlog, KBS_CAPACITY)
    print(f"{minute:3}  {'up  ' if up else 'down'} {failed:12,}"
          f" {backlog:22,}")

min  KBS   sync: failed so far   async: waiting in queue
  0  up              0                      0
  1  up              0                      0
  2  up              0                      0
  3  down        1,000                  1,000
  4  down        2,000                  2,000
  5  down        3,000                  3,000
  6  up          3,000                  1,500
  7  up          3,000                      0
  8  up          3,000                      0
  9  up          3,000                      0

With synchronous calls, 3,000 readings fail and have to be resent by the meters, or are lost. With a queue, nothing fails: the readings wait, and KBS catches up within two minutes after it is back, because it can process faster than readings arrive. A queue removes temporal coupling; it does not remove the coupling to the message format, which is why API Design Between Services treats message schemas like any other contract.

Remote calls are not function calls

A call to another service looks like a function call in the code, which makes it easy to forget everything that can go wrong in between. Engineers at Sun Microsystems summed up the false assumptions in the 1990s as the fallacies of distributed computing (Peter Deutsch listed seven, James Gosling added the eighth): the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn't change; there is one administrator; transport cost is zero; the network is homogeneous. Each one is a bug waiting to happen.

The most common one in practice is the first. A request can fail before it reaches the server, the server can fail while processing it, or the response can be lost on the way back. In the last case the caller sees an error although the work was done. The caller cannot tell these cases apart, which has two consequences:

• Every remote call needs a timeout. Without one, a hung service holds the caller's threads and connections forever. Python is a good example of why this is easy to miss: unless you pass timeout=, sockets have none.
• Retrying is only safe for idempotent operations. If the first attempt actually succeeded, a retry repeats it. That is harmless for reading usage data and harmful for charging a customer, unless the operation is made idempotent, for example with an idempotency key.

import socket

# The default timeout for new sockets, used by urllib, http.client and
# most libraries built on them unless you pass timeout= yourself.
print("default socket timeout:", socket.getdefaulttimeout())

default socket timeout: None

None means "wait forever": a urllib.request.urlopen() call without timeout= waits as long as the operating system keeps the connection open. How to choose timeout values, how to retry with backoff and jitter, and when to stop calling a failing service with a circuit breaker is explained on Reliability Patterns.

Message brokers and queues

A message broker is software that receives messages from the services that produce them, stores them safely and delivers them to the services that consume them, so that producers and consumers do not need to know about each other or be available at the same time. A message queue is a buffer, kept by a broker or a cloud service, that holds messages in order until a consumer takes one and acknowledges it; each message is handled by one consumer. (The data structure behind it is the ordinary queue: first in, first out.)

Brokers support two basic ways of delivering:

• Point-to-point: a message on a queue goes to one consumer. Several instances of the same consumer can read from one queue to share the work; Microsoft's Azure Architecture Center calls this the competing consumers pattern.
• Publish-subscribe: a message goes to every interested subscriber, each with its own copy. That is the basis of Event-Driven Architecture.

A queue also smooths out bursts. The Azure catalog calls it queue-based load leveling: "Use a queue that creates a buffer between a task and a service to smooth intermittent heavy loads." Kestrel's meters do not report evenly: most report in the first minute after :00, :15, :30 and :45. If KBS processes 1,000 readings per second from a queue:

# 400,000 meters report within the first 60 seconds after :00.
# KBS takes readings from the queue at its own steady pace.
READINGS, BURST_SECONDS = 400_000, 60
CONSUME_PER_SECOND = 1_000


def arriving(second):
    """Readings arriving in this second (spread evenly over the burst)."""
    if second >= BURST_SECONDS:
        return 0
    return READINGS * (second + 1) // BURST_SECONDS - READINGS * second // BURST_SECONDS


queue, second, peak = 0, 0, 0
while second < BURST_SECONDS or queue:
    queue += arriving(second)
    queue -= min(queue, CONSUME_PER_SECOND)
    peak = max(peak, queue)
    second += 1
    if second in (30, 60, 120, 240):
        print(f"after {second:3} s: {queue:7,} readings waiting")

print(f"peak {peak:,} waiting; queue empty after {second} s"
      f" ({second // 60} min {second % 60} s)")

after  30 s: 170,000 readings waiting
after  60 s: 340,000 readings waiting
after 120 s: 280,000 readings waiting
after 240 s: 160,000 readings waiting
peak 340,000 waiting; queue empty after 400 s (6 min 40 s)

KBS works at a steady 1,000 readings per second while the burst arrives at more than six times that rate, and the queue is empty long before the next quarter hour. The price is delay: the last reading of the burst reaches KBS almost seven minutes after it was sent. For billing that is fine; for an alarm it would not be. (Little's law and queueing delay are covered on Capacity Planning and Performance.)

The best-known products, as of October 2026:

Delivery guarantees

A broker and a consumer agree that a message is done when the consumer acknowledges it. When exactly that happens decides what can go wrong. The Apache Kafka documentation names the possibilities:

• At-most-once delivery: each message is delivered once or not at all; messages may be lost but are never delivered twice. This is what you get when the consumer acknowledges a message as soon as it receives it, before processing.
• At-least-once delivery: no message is lost, but some may be delivered more than once. This is what you get when the consumer acknowledges only after processing: if it crashes in between, the broker delivers the message again.
• Exactly once: "Each message is processed once and only once." This is what everybody wants, and the next paragraphs explain why it needs help from the consumer.

A consumer that crashes once, on reading R2, in both modes:

from collections import deque


def consume(mode, crash_on):
    """Take readings from a queue; the consumer crashes once, on `crash_on`."""
    queue = deque(["R1", "R2", "R3"])
    processed, crashed = [], False
    while queue:
        message = queue[0]
        if mode == "at-most-once":
            queue.popleft()                  # acknowledge first ...
            if message == crash_on and not crashed:
                crashed = True               # ... crash while processing: lost
                continue
            processed.append(message)
        else:
            processed.append(message)        # process first ...
            if message == crash_on and not crashed:
                crashed = True               # ... crash before the ack:
                continue                     # the broker delivers it again
            queue.popleft()                  # ... then acknowledge
    return processed


for mode in ("at-most-once", "at-least-once"):
    print(f"{mode:13}  processed: {' '.join(consume(mode, 'R2'))}")

at-most-once   processed: R1 R3
at-least-once  processed: R1 R2 R2 R3

At most once loses R2; at least once processes it twice. Most systems choose at least once, because a duplicate can be detected and a lost message cannot. The RabbitMQ documentation states it directly: "Use of acknowledgements guarantees at least once delivery. Without acknowledgements, message loss is possible".

Exactly-once processing means that the effect of each message is applied exactly once, even though the message itself may be delivered more than once. It is built from at-least-once delivery plus a consumer that recognizes duplicates or applies its changes in the same transaction that records its progress. That is why this page says processing and not delivery: across an unreliable network, no system can guarantee that a message travels exactly once. Kafka supports exactly-once semantics within Kafka itself (Kafka Streams and transactions), but its documentation adds that exactly-once for other destinations "generally requires cooperation with such systems".

Idempotent consumers

An idempotent consumer is a message handler that produces the same result whether it processes a given message once or several times, usually by recording which messages it has already handled and skipping duplicates. microservices.io describes the usual implementation: "record the IDs of processed messages in the database" and "detect and discard duplicates". The serverless page shows the basic version with a set of message IDs.

Two details make the difference in practice. First, the duplicate check and the effect must be committed together, in one transaction; otherwise a crash between them still produces a duplicate or a loss. Second, the message ID is not always the right key. When a smart meter gets no answer in time, it sends the same reading again, and meter-api publishes it as a new message with a new ID. A business key catches that case: one meter (identified by its EAN code) has one reading per 15-minute interval.

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE usage (ean TEXT, interval_start TEXT, kwh REAL,
                    PRIMARY KEY (ean, interval_start));
CREATE TABLE totals (ean TEXT PRIMARY KEY, kwh REAL);
""")

EAN = "871687400000000017"
deliveries = [
    ("m-101", EAN, "2026-10-06T02:15", 0.42),
    ("m-102", EAN, "2026-10-06T02:30", 0.38),
    ("m-102", EAN, "2026-10-06T02:30", 0.38),  # broker redelivers m-102
    ("m-117", EAN, "2026-10-06T02:30", 0.38),  # meter resent it: new message ID
]


def handle(msg_id, ean, interval_start, kwh):
    with db:  # the duplicate check and the effect commit together, or not at all
        inserted = db.execute(
            "INSERT INTO usage VALUES (?, ?, ?) ON CONFLICT DO NOTHING",
            (ean, interval_start, kwh)).rowcount
        if not inserted:
            return "duplicate, skipped"
        db.execute("INSERT INTO totals VALUES (?, ?) ON CONFLICT (ean) "
                   "DO UPDATE SET kwh = kwh + excluded.kwh", (ean, kwh))
        return "stored"


for message in deliveries:
    print(f"{message[0]}  {message[2]}  {handle(*message)}")
(total,) = db.execute("SELECT kwh FROM totals WHERE ean = ?", (EAN,)).fetchone()
print(f"total for the meter: {total:.2f} kWh")

m-101  2026-10-06T02:15  stored
m-102  2026-10-06T02:30  stored
m-102  2026-10-06T02:30  duplicate, skipped
m-117  2026-10-06T02:30  duplicate, skipped
total for the meter: 0.80 kWh

The redelivered m-102 and the resent m-117 are both recognized, because the primary key on the meter and interval refuses a second row, and the total is updated in the same transaction. Where no natural key exists, store the message ID in a table with a unique constraint instead, and clean old IDs up after a period longer than the broker can redeliver.

When a message keeps failing

Some messages can never be processed: a reading with "kwh": "n/a", an event for a customer that does not exist, a message in an old format. Retried forever, such a poison message blocks the queue or wastes resources. A dead-letter queue (DLQ) is a separate queue to which a broker or consumer moves messages that could not be processed after a set number of attempts, so that they stop blocking other work and can be inspected, fixed and replayed by a person.

MAX_ATTEMPTS = 3
queue = [{"id": "R7", "kwh": "0.41"}, {"id": "R8", "kwh": "n/a"},
         {"id": "R9", "kwh": "0.36"}]
attempts, dead_letters = {}, []


def process(message):
    return float(message["kwh"])     # raises ValueError for "n/a"


while queue:
    message = queue.pop(0)
    attempts[message["id"]] = attempts.get(message["id"], 0) + 1
    try:
        kwh = process(message)
        print(f"{message['id']}: stored {kwh} kWh")
    except ValueError as error:
        n = attempts[message["id"]]
        if n < MAX_ATTEMPTS:
            print(f"{message['id']}: attempt {n} failed ({error}), retry later")
            queue.append(message)
        else:
            print(f"{message['id']}: attempt {n} failed, moved to dead-letter queue")
            dead_letters.append(message)

print("dead-letter queue:", [m["id"] for m in dead_letters])

R7: stored 0.41 kWh
R8: attempt 1 failed (could not convert string to float: 'n/a'), retry later
R9: stored 0.36 kWh
R8: attempt 2 failed (could not convert string to float: 'n/a'), retry later
R8: attempt 3 failed, moved to dead-letter queue
dead-letter queue: ['R8']

R9 is processed while R8 waits for its next attempt, and after three attempts R8 is set aside instead of being retried forever. Most brokers and cloud queues do this for you with a setting for the maximum number of deliveries. Two rules make a DLQ useful instead of a silent trash can:

• Alert on it. A message in the DLQ is data that was not processed: here, a reading that will be missing from a customer's bill. Alert when the DLQ is not empty, and give it an owner.
• Plan the replay. After a fix, messages must be sent through again, which is only safe because the consumer is idempotent.

Ordering

A queue is first in, first out, but the order in which messages are processed is still not guaranteed in general. Competing consumers work in parallel, a redelivered message comes back after newer ones, and the DLQ example above processed R9 before R8. Kafka keeps order only within one partition, so messages that must stay in order need the same partition key, such as the meter's EAN code.

The robust approach is to design consumers that do not depend on order: include the time or a version number in each message, and let the consumer ignore a message that is older than what it already has. A correction to the reading for 02:30 should say which interval it corrects, not rely on arriving after the original.

Choosing a style per interaction

Most systems use both styles. The question is asked per interaction, not per system:

Kestrel uses both, each where it fits:

• Readings to billing: asynchronous. Nobody waits for KBS to book a reading, the load comes in bursts, and KBS must not be able to make meters fail. After INC-22047 the team proposed this in ADR-0007, an architecture decision record that is still under review (see Documenting Architecture), with a transactional outbox (see Event-Driven Architecture).
• "Show my usage" in the portal: synchronous. A customer is waiting for the graph. The portal calls meter-api with an 800 ms timeout, and when that runs out it shows the account without the graph and a "usage data is delayed" notice: graceful degradation.