How a Kubernetes Cluster Works

The big picture: everything goes through the API

Kubernetes runs containers across a group of machines and keeps them running. The overview on Kubernetes basics introduces the parts of a cluster; this page shows how they work together, because that is what you need when something does not happen as expected.

The design rests on three ideas:

• One API. Every object — a pod, a deployment, a node — is a record behind one REST API (representational state transfer, see REST APIs). kubectl, a CI/CD pipeline and the cluster's own components all use that same API.
• Declared state. You store what you want (three copies of the portal, image 1.4.2); you do not issue a sequence of commands. See declarative versus imperative.
• Independent watchers. Each component watches the API for changes in the objects it cares about and does its own small job. No component orders another one around.

For troubleshooting this has a practical consequence: when a pod does not start, the question is always which watcher did not do its part. Did a controller not create the pod, did the scheduler not find a node, or did the kubelet fail to start the container? The events that each component writes tell you, as the walk-through below shows.

The control plane

The control plane is the set of components that store the cluster state and make the decisions: the API server, etcd, the scheduler and the controller manager (plus, in a cloud, the cloud controller manager that talks to the provider's API for load balancers and disks). It runs no application containers of its own in a normal production cluster. With managed Kubernetes (EKS, AKS, GKE; see managed Kubernetes) the provider runs the control plane and you never log on to it.

The name is borrowed from networking, where the control plane decides and the data plane forwards packets. In Kubernetes the "data plane" is the worker nodes, where the containers run.

The API server

The API server (kube-apiserver) is the front door of the cluster and the only component that talks to etcd. Every request passes the same steps:

1. Authentication: who is this — a user with a client certificate or token, or a service account?
2. Authorization: may this identity do this verb on this resource in this namespace? Usually decided by RBAC (role-based access control), explained on namespaces, RBAC and Kubernetes security.
3. Admission: admission controllers may change the object (add defaults) or reject it (for example a pod that breaks the namespace's security level).
4. Validation and storage: the object is checked against its schema and written to etcd.

The API server then notifies everyone who is watching that kind of object. Watches are long-lived HTTPS (Hypertext Transfer Protocol Secure) connections, which is how the other components hear about changes within milliseconds without polling.

etcd

etcd is a distributed, strongly consistent key-value store that holds every object of the cluster. Its members agree on each write with the Raft consensus algorithm: a write only counts when a majority (the quorum) has stored it. That rule prevents split-brain, and it also explains the member counts you see everywhere:

for members in range(1, 8):
    quorum = members // 2 + 1        # a strict majority
    tolerated = members - quorum     # members that may fail
    print(f"{members} members: quorum {quorum}, survives {tolerated} failure(s)")

1 members: quorum 1, survives 0 failure(s)
2 members: quorum 2, survives 0 failure(s)
3 members: quorum 2, survives 1 failure(s)
4 members: quorum 3, survives 1 failure(s)
5 members: quorum 3, survives 2 failure(s)
6 members: quorum 4, survives 2 failure(s)
7 members: quorum 4, survives 3 failure(s)

Four members survive no more failures than three, but need one more vote per write and give one more machine that can break. Production clusters therefore run three or five etcd members, each on a separate machine and ideally in a separate failure zone.

The scheduler

The kube-scheduler watches for pods that have no node yet. For each one it first filters the nodes (enough free CPU and memory for the pod's resource requests, matching node selectors, tolerated taints, a usable volume zone) and then scores the remaining nodes (spread replicas, prefer nodes that already have the image). It writes the chosen node name into the pod — that is all. It does not start anything.

If no node passes the filters, the pod stays Pending with a FailedScheduling event that lists why each node was rejected. Note that this is a different "scheduler" from the operating system's CPU scheduler, which divides processor time between processes on one machine.

The controller manager

kube-controller-manager is one program that runs dozens of controllers: for Deployments, ReplicaSets, Jobs, nodes, service accounts, EndpointSlices and more. They are bundled into one process for convenience; each still works independently. Controllers are explained below.

Worker nodes

A node is a machine — a virtual machine or a physical server — that is registered with the cluster and runs pods. Every node runs three pieces of software.

The kubelet

The kubelet is the agent on each node. It watches the API server for pods assigned to its node, asks the container runtime to pull images and start containers, mounts volumes, runs the probes and reports pod status and node health back to the API server. When a node stops reporting (after a grace period of under a minute without a heartbeat), the control plane marks it NotReady and eventually moves its pods elsewhere.

The kubelet is a normal systemd service on the node, not a container. When a node is NotReady, the kubelet's own log (journalctl -u kubelet) is where you look.

The container runtime

The kubelet does not run containers itself. It talks to a container runtime through the CRI (Container Runtime Interface), a gRPC API. The common runtimes are containerd and CRI-O; both run standard OCI images (see what are containers?). Kubernetes removed its built-in Docker Engine adapter, the dockershim, in version 1.24. Images built with docker build still run everywhere — only Docker Engine itself is no longer the runtime on nodes. Kubernetes 1.35 was the last release that supports containerd 1.x; newer releases need containerd 2.

On a node, crictl is the runtime's command-line tool: crictl ps -a lists containers, crictl logs shows their output. You rarely need it, except when the control plane itself is broken and kubectl has nothing to talk to.

kube-proxy and the network plugin

kube-proxy programs packet-forwarding rules (iptables or nftables) on each node so that traffic to a Service address reaches one of its pods. Some network plugins replace it entirely. Its IPVS mode, once recommended for performance, has been deprecated since Kubernetes 1.35. The network plugin (CNI, Container Network Interface: Calico, Cilium, Flannel, a cloud's own plugin) gives every pod an IP address and connects pods across nodes.

Controllers and desired state

A controller is a control loop that watches one kind of object and keeps the actual state in line with the object's spec, the desired state. It writes what it observed into the object's status. The idea is the reconciliation loop, simulated in Python on Kubernetes basics.

Controllers chain: one controller's output is another controller's input.

• The Deployment controller sees a Deployment and creates or updates a ReplicaSet.
• The ReplicaSet controller sees a ReplicaSet that wants three pods and finds none, so it creates three pod objects.
• The scheduler sees three pods without a node and assigns them.
• The kubelet on each chosen node sees a pod for itself and starts it.

Controllers are level-based: they act on the current difference, not on a missed event. If a controller restarts or misses a notification, its next pass still sees "three wanted, two exist" and fixes it. This is also why manual fixes do not stick: delete a pod by hand and the ReplicaSet controller creates a replacement; edit a live object that a GitOps tool manages and that tool changes it back (see configuration drift).

From kubectl apply to a running container

Ravi Kumar, tech lead of Kestrel Energy's customer portal, applies a Deployment with three replicas of registry.example.com/kestrel/portal:1.4.2 to the namespace kestrel-portal. This is what happens, in order:

1. kubectl reads the file, finds the cluster and credentials in its kubeconfig and sends the Deployment to the API server.
2. The API server authenticates, authorizes, runs admission, validates and stores the Deployment in etcd. kubectl prints deployment.apps/portal created and is done — nothing is running yet.
3. The Deployment controller creates a ReplicaSet; the ReplicaSet controller creates three pod objects (see pods, Deployments and ReplicaSets).
4. The kube-scheduler picks a node for each pod and records it.
5. The kubelet on each chosen node sees its new pod.
6. The kubelet asks the runtime to pull the image and start the container, then reports the status back: ContainerCreating, then Running, then ready once the readiness probe passes.

Each step leaves an event, so the chain is visible afterward:

$ kubectl get events -n kestrel-portal --sort-by=.metadata.creationTimestamp

LAST SEEN   TYPE     REASON              OBJECT                         MESSAGE
41s         Normal   ScalingReplicaSet   deployment/portal              Scaled up replica set portal-7c9d8b6f4d from 0 to 3
41s         Normal   SuccessfulCreate    replicaset/portal-7c9d8b6f4d   Created pod: portal-7c9d8b6f4d-x2v9k
41s         Normal   Scheduled           pod/portal-7c9d8b6f4d-x2v9k    Successfully assigned kestrel-portal/portal-7c9d8b6f4d-x2v9k to kestrel-worker2
40s         Normal   Pulling             pod/portal-7c9d8b6f4d-x2v9k    Pulling image "registry.example.com/kestrel/portal:1.4.2"
37s         Normal   Pulled              pod/portal-7c9d8b6f4d-x2v9k    Successfully pulled image "registry.example.com/kestrel/portal:1.4.2" in 2.817s (2.817s including waiting). Image size: 48213042 bytes.
…

Read it as a chain of hand-overs. If the list stops after SuccessfulCreate, the scheduler could not place the pod (look for FailedScheduling). If it stops after Pulling, the image could not be fetched. If there is not even a ScalingReplicaSet, the Deployment itself was rejected or is in another namespace. Each of these failures is worked out on kubectl and troubleshooting.

The cluster's own pods

In many clusters — those built with kubeadm, kind and most self-managed installers — the control-plane components themselves run as pods in the kube-system namespace. They are static pods: the kubelet on the control-plane node starts them from manifest files in /etc/kubernetes/manifests, so they exist even before the API server is up. On a three-node test cluster:

$ kubectl get nodes
$ kubectl get pods -n kube-system

NAME                    STATUS   ROLES           AGE     VERSION
kestrel-control-plane   Ready    control-plane   3m12s   v1.37.0
kestrel-worker          Ready    <none>          2m48s   v1.37.0
kestrel-worker2         Ready    <none>          2m48s   v1.37.0

NAME                                            READY   STATUS    RESTARTS   AGE
coredns-7c65d6cfc9-4xq2b                        1/1     Running   0          3m4s
coredns-7c65d6cfc9-vh8nm                        1/1     Running   0          3m4s
etcd-kestrel-control-plane                      1/1     Running   0          3m12s
kindnet-6tq9r                                   1/1     Running   0          2m48s
kindnet-dk2zl                                   1/1     Running   0          2m48s
kindnet-p8w7c                                   1/1     Running   0          3m4s
kube-apiserver-kestrel-control-plane            1/1     Running   0          3m12s
kube-controller-manager-kestrel-control-plane   1/1     Running   0          3m12s
kube-proxy-5jz8f                                1/1     Running   0          2m48s
kube-proxy-9mbkx                                1/1     Running   0          3m4s
kube-proxy-tl6hv                                1/1     Running   0          2m48s
kube-scheduler-kestrel-control-plane            1/1     Running   0          3m12s

Static pods carry the node name (etcd-kestrel-control-plane). kube-proxy and the network plugin (kindnet here) run once per node, so there are three of each. CoreDNS provides the cluster DNS. On managed Kubernetes you see the per-node pods and CoreDNS, but not the control plane: the provider runs it out of sight.

Versions and upgrades

Kubernetes releases a new minor version about three times a year, and each minor version gets roughly a year of patch releases. As of October 2026:

Two rules matter in daily work:

• Upgrade one minor version at a time (1.35 → 1.36 → 1.37), control plane first, then the nodes. Read the release notes of each step for removed APIs; old manifests that use a removed apiVersion are rejected after the upgrade.
• kubectl may differ by one minor version from the API server, older or newer. A kubectl 1.37 against a 1.35 cluster is unsupported and can fail in confusing ways.

Amara Okafor on the service desk keeps a list of Kestrel's clusters. This check tells her which ones need an upgrade ticket and whether the team's kubectl fits:

from datetime import date

TODAY = date(2026, 10, 6)
END_OF_LIFE = {  # end of patch support per minor release (kubernetes.io/releases)
    "1.37": date(2027, 10, 28),
    "1.36": date(2027, 6, 28),
    "1.35": date(2027, 2, 28),
    "1.34": date(2026, 10, 27),
    "1.33": date(2026, 6, 28),
}


def minor(version):
    """'v1.35.9' -> '1.35'"""
    parts = version.lstrip("v").split(".")
    return f"{parts[0]}.{parts[1]}"


def check(cluster, kubectl):
    eol = END_OF_LIFE[minor(cluster)]
    days = (eol - TODAY).days
    support = f"{days} days of support left" if days > 0 else "END OF LIFE"
    skew = abs(int(minor(cluster).split(".")[1]) - int(minor(kubectl).split(".")[1]))
    client = "kubectl ok" if skew <= 1 else f"kubectl {kubectl} too far apart"
    print(f"{cluster:9} {support:26} {client}")


check("v1.37.1", "v1.37.1")
check("v1.35.9", "v1.36.5")
check("v1.35.9", "v1.37.1")
check("v1.34.12", "v1.34.12")
check("v1.33.7", "v1.34.12")

v1.37.1   387 days of support left   kubectl ok
v1.35.9   145 days of support left   kubectl ok
v1.35.9   145 days of support left   kubectl v1.37.1 too far apart
v1.34.12  21 days of support left    kubectl ok
v1.33.7   END OF LIFE                kubectl ok

Trying it locally

To learn, you do not need a cloud account. Tools such as kind (Kubernetes in Docker), minikube and k3s run a small cluster on a laptop. kind runs each "node" as a Docker container; it is built for testing Kubernetes itself and for CI (continuous integration) pipelines, and is handy for local experiments. As of October 2026 the current release is kind v0.33.0, which creates Kubernetes 1.37.0 nodes by default. A configuration with one control-plane node and two workers:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
networking:
  apiServerAddress: "127.0.0.1"   # API server reachable from this machine only

$ kind create cluster --name kestrel --config kind-config.yaml
$ kubectl version --client

Client Version: v1.37.1
Kustomize Version: v5.8.1

kind create cluster writes the new cluster's address and credentials into your kubeconfig and makes it the current context, so the next kubectl command talks to it. Check which cluster you are talking to (kubectl config current-context) before every command that changes something; mixing up a test cluster and production is a classic mistake.