ConfigMaps, Secrets and Configuration

Configuration outside the image

Kestrel Energy builds the meter-api image once and runs exactly that image in test and in production. What differs between the two is configuration: the log level, the database host, the database password. If those were baked into the image, every environment would need its own build, and the production password would sit in every copy of the image and in the registry.

Kubernetes keeps configuration in separate objects that the pod refers to. The overview on Kubernetes basics introduces them; this page covers how they behave in practice:

• a ConfigMap for ordinary settings;
• a Secret for values that must stay confidential;
• both delivered as environment variables or as files in a volume.

ConfigMaps

A ConfigMap is a Kubernetes object that stores non-confidential configuration as key-value pairs, where a value can be a short string or the contents of a whole file. Its data cannot exceed 1 MiB (mebibyte). kubectl can create one from literals or files:

$ kubectl create configmap meter-api-config --from-literal=LOG_LEVEL=info --from-literal=TARIFF_REGION=NL --dry-run=client -o yaml

apiVersion: v1
data:
  LOG_LEVEL: info
  TARIFF_REGION: NL
kind: ConfigMap
metadata:
  name: meter-api-config

In the repository, the ConfigMap also carries a small settings file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: meter-api-config
  namespace: kestrel-portal
data:
  LOG_LEVEL: info
  TARIFF_REGION: NL
  settings.toml: |
    [meter]
    batch_size = 500
    retry_seconds = 30

The 1 MiB limit is there because every object lives in etcd, which is built for many small records, not for files. Joost Bakker wanted to put the complete tariff table into a ConfigMap; a quick size check, counting the UTF-8 bytes of all keys and values, settles it:

LIMIT = 1024 * 1024  # 1 MiB = 1,048,576 bytes


def data_size(data):
    """Bytes of all keys and values, UTF-8 encoded."""
    return sum(len(k.encode()) + len(v.encode()) for k, v in data.items())


small = {
    "LOG_LEVEL": "info",
    "TARIFF_REGION": "NL",
    "settings.toml": '[meter]\nbatch_size = 500\nretry_seconds = 30\n',
}
# 50,000 lines of a tariff table, generated so the example needs no file
tariffs = "".join(f"NL-{n:05d};0.2731;0.0954\n" for n in range(50_000))
large = dict(small, **{"tariffs.csv": tariffs})

for name, data in [("meter-api-config", small), ("meter-api-tariffs", large)]:
    size = data_size(data)
    verdict = "fits" if size <= LIMIT else "TOO LARGE"
    print(f"{name:18} {size:>9,} bytes  {size / LIMIT:6.1%} of 1 MiB  {verdict}")

meter-api-config          85 bytes    0.0% of 1 MiB  fits
meter-api-tariffs  1,150,096 bytes  109.7% of 1 MiB  TOO LARGE

Large data belongs in a database, in object storage or in the image (if it changes with the code), not in a ConfigMap. Setting immutable: true on a ConfigMap that never changes protects it against accidental edits and saves the kubelets work, because they no longer watch it; to change it you create a new one under a new name.

Using configuration in a pod

A pod refers to ConfigMaps and Secrets by name; they must exist in the same namespace before the pod starts, or the pod waits with an error event. Three ways to use them, all in one pod template:

spec:
  containers:
    - name: meter-api
      image: registry.example.com/kestrel/meter-api:2.3.0
      envFrom:
        - configMapRef:
            name: meter-api-config          # every key becomes a variable
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: meter-db                # one key from a Secret
              key: DB_PASSWORD
      volumeMounts:
        - name: config
          mountPath: /etc/meter-api         # files: /etc/meter-api/settings.toml …
          readOnly: true
  volumes:
    - name: config
      configMap:
        name: meter-api-config
        items:
          - key: settings.toml
            path: settings.toml

For secrets, files are the safer choice: an environment variable is inherited by every child process, shows up in some debugging tools, and is a classic source of leaks when an application logs its whole environment at start-up.

When changes arrive

• Environment variables are never updated in a running container. The new values arrive only with new pods.
• Mounted files are updated eventually: the kubelet syncs them periodically, so a change takes from seconds up to about a minute or more, depending on its sync period and cache.
• Files mounted with subPath are never updated, because a subPath mount is a copy of one file at start.

To make every pod pick up a changed ConfigMap or Secret, restart the Deployment; it performs a normal, gradual rollout:

$ kubectl rollout restart deployment/meter-api -n kestrel-portal

deployment.apps/meter-api restarted

Projected volumes

A projected volume combines several sources — ConfigMaps, Secrets, information about the pod itself (the downward API) and a short-lived service account token — into one directory. The application then finds everything it needs under a single path:

volumes:
  - name: app-config
    projected:
      sources:
        - configMap:
            name: meter-api-config
        - secret:
            name: meter-db
        - downwardAPI:
            items:
              - path: pod-name
                fieldRef:
                  fieldPath: metadata.name
        - serviceAccountToken:
            path: token
            audience: vault.example.com
            expirationSeconds: 3600

Kubernetes itself uses a projected volume to give every pod its service account token. That token, unlike the old long-lived tokens stored in Secrets, expires and is renewed by the kubelet (see Kubernetes service accounts).

Secrets

A Secret is a Kubernetes object for small confidential values such as passwords, tokens, keys and certificates, used in pods like a ConfigMap. Its values are stored base64-encoded, its size is limited to 1 MiB, and it is only as protected as the cluster's encryption and access rules make it. Kubernetes treats Secrets somewhat more carefully than ConfigMaps — the kubelet keeps them in memory (tmpfs) on the node rather than on disk, and kubectl describe shows only their sizes — but by default that is all.

$ kubectl create secret generic meter-db --from-literal=DB_PASSWORD=example-only --dry-run=client -o yaml

apiVersion: v1
data:
  DB_PASSWORD: ZXhhbXBsZS1vbmx5
kind: Secret
metadata:
  name: meter-db

Secrets have a type that tells tools what keys to expect. The common ones:

In a manifest you may write plain text under stringData; the API server encodes it into data when it stores the object. That makes the next problem obvious: a Secret manifest is a file with the password in it.

apiVersion: v1
kind: Secret
metadata:
  name: meter-db
  namespace: kestrel-portal
type: Opaque
stringData:
  DB_USER: meter_api
  DB_PASSWORD: example-only-not-a-real-secret   # never a real value in Git

base64 is encoding, not encryption

Base64 turns bytes into 64 printable characters so that any value, including binary data such as a key file, fits in YAML or JSON. It uses no key. Reversing it takes one command:

echo ZXhhbXBsZS1vbmx5 | base64 -d; echo

example-only

The same in Python, for a Secret's complete data field. Note also that encoding the same value twice gives the same text both times; real encryption depends on a key, and good schemes produce different output each time:

import base64

# The data field of a Secret, exactly as "kubectl get secret -o yaml" shows it
secret_data = {
    "DB_PASSWORD": "ZXhhbXBsZS1vbmx5",
    "DB_USER": "bWV0ZXJfYXBp",
}
for key, value in secret_data.items():
    print(f"{key:12} {value:18} -> {base64.b64decode(value).decode()}")

# Encoding uses no key: anyone gets the same result, every time
print(base64.b64encode(b"example-only").decode())
print(base64.b64encode(b"example-only").decode())

DB_PASSWORD  ZXhhbXBsZS1vbmx5   -> example-only
DB_USER      bWV0ZXJfYXBp       -> meter_api
ZXhhbXBsZS1vbmx5
ZXhhbXBsZS1vbmx5

In a cluster, anyone allowed to read the Secret gets the password the same way:

$ kubectl get secret meter-db -n kestrel-portal -o yaml
$ kubectl get secret meter-db -n kestrel-portal -o jsonpath='{.data.DB_PASSWORD}' | base64 -d; echo

apiVersion: v1
data:
  DB_PASSWORD: ZXhhbXBsZS1vbmx5
kind: Secret
metadata:
  creationTimestamp: "2026-09-30T08:12:44Z"
  name: meter-db
  namespace: kestrel-portal
  resourceVersion: "4182736"
  uid: 3f6b2c9e-8d41-4a7e-9b0c-5e2d7a1f6c83
type: Opaque

example-only

How Secrets are protected — and how not

By default, upstream Kubernetes gives Secrets far less protection than the name suggests:

• They are stored unencrypted in etcd. Anyone with access to etcd, or to an etcd backup, can read them all.
• Anyone who may read Secrets (get, or list — which returns the values too) in a namespace can read them through the API.
• Anyone who may create pods in a namespace can read every Secret there, by starting a pod that mounts it and prints it. This is the least obvious path, and the reason why "create pods" in a production namespace is a powerful permission.

The Kubernetes documentation lists the countermeasures; they work together.

Encryption at rest

The API server can encrypt Secrets before writing them to etcd (encryption at rest). Use a KMS provider, so that the key encryption key lives in a key management service outside the cluster, rather than a key in a file next to the data. On a self-managed control plane this is a configuration file for the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: kestrel-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}       # still able to read old, unencrypted data

Existing Secrets are only encrypted when they are written again, so after enabling it all Secrets have to be rewritten once. On managed Kubernetes, encryption is a setting of the service (and some services enable it by default); check your cluster rather than assuming. Encryption at rest protects etcd and its backups — it does nothing against someone who is allowed to read the Secret through the API.

Limiting access

• Grant get, list and watch on Secrets only to the identities that need them, per namespace, never cluster-wide by habit. RBAC (role-based access control) and service accounts are covered on namespaces, RBAC and Kubernetes security.
• Treat permission to create pods in a namespace as permission to read its Secrets.
• Mount a Secret only into the container that needs it, not into every container of the pod.
• Do not log environment variables or configuration at start-up; mask secret values in error reports.

External secret managers

The strongest pattern keeps the real value out of the cluster's manifests entirely. It lives in a secrets manager — HashiCorp Vault or OpenBao, AWS Secrets Manager, Azure Key Vault, Google Secret Manager — where it is audited and rotated, and the cluster fetches it.

• The External Secrets Operator is an open-source Kubernetes controller that reads values from an external secrets manager and writes them into ordinary Kubernetes Secrets, refreshing them on a schedule. Applications keep using normal Secrets; Git holds only a reference.
• The Secrets Store CSI Driver mounts values from the secrets manager directly into the pod as files, without (necessarily) creating a Kubernetes Secret.
• Sealed Secrets and SOPS take a different route: the value is encrypted with a key that only the cluster or the deployment tool holds, so the encrypted file can safely be committed. See secrets in GitOps.

An ExternalSecret that keeps the meter-db Secret filled from the company vault:

apiVersion: external-secrets.io/v1      # check the version your installation serves
kind: ExternalSecret
metadata:
  name: meter-db
  namespace: kestrel-portal
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: kestrel-vault
  target:
    name: meter-db                        # the Kubernetes Secret it creates
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: kestrel/meter-api/db
        property: password

The result is still a Kubernetes Secret, so encryption at rest and RBAC still matter. What changes is that the value has one owner and one place to rotate it: Hamid Rahimi, the DBA, rotates the database password in the vault, and within the refresh interval every cluster has the new value — after which the pods still need a restart if they read it from an environment variable.

Keeping secrets out of Git

Secret scanning on the repository catches many known token formats, but not a database password that looks like any other string. A simple check in the pipeline adds a rule for manifests: no Secret with values, and no sensitive-looking variable with a literal value. A sketch of such a check, on objects as a YAML parser would return them:

# Objects as a YAML parser would return them from the files in a Git repository
manifests = [
    ("deploy/meter-api.yaml", {
        "kind": "Deployment",
        "metadata": {"name": "meter-api"},
        "spec": {"template": {"spec": {"containers": [{
            "name": "meter-api",
            "env": [
                {"name": "LOG_LEVEL", "value": "info"},
                {"name": "API_TOKEN", "value": "example-only-not-a-real-secret"},
                {"name": "DB_PASSWORD", "valueFrom": {"secretKeyRef": {"name": "meter-db", "key": "DB_PASSWORD"}}},
            ],
        }]}}},
    }),
    ("deploy/meter-db-secret.yaml", {
        "kind": "Secret",
        "metadata": {"name": "meter-db"},
        "data": {"DB_PASSWORD": "ZXhhbXBsZS1vbmx5"},
    }),
    ("deploy/meter-api-config.yaml", {
        "kind": "ConfigMap",
        "metadata": {"name": "meter-api-config"},
        "data": {"LOG_LEVEL": "info"},
    }),
]

SENSITIVE = ("PASSWORD", "TOKEN", "SECRET", "KEY")
findings = []
for path, obj in manifests:
    name = f'{obj["kind"]}/{obj["metadata"]["name"]}'
    if obj["kind"] == "Secret" and (obj.get("data") or obj.get("stringData")):
        keys = sorted({**obj.get("data", {}), **obj.get("stringData", {})})
        findings.append(f"{path}: {name} contains values for {', '.join(keys)}")
    for container in obj.get("spec", {}).get("template", {}).get("spec", {}).get("containers", []):
        for var in container.get("env", []):
            if "value" in var and any(word in var["name"] for word in SENSITIVE):
                findings.append(f'{path}: {name} sets {var["name"]} as plain text')

print("\n".join(findings) if findings else "no findings")

deploy/meter-api.yaml: Deployment/meter-api sets API_TOKEN as plain text
deploy/meter-db-secret.yaml: Secret/meter-db contains values for DB_PASSWORD

The DB_PASSWORD that uses secretKeyRef passes: it is a reference, not a value. The fix for both findings is the same — move the value into the vault and reference it.